Fake server test : contd

The error raised in the PROPFIND block could fixed with the help of Philipp. It was due to wrong sequence of initialization messages to the calendar. To initialize a CalDav calendar properly, a sequence of asynchronous messages need to be sent.

  1. Initial PROPFIND response
  2. OPTIONS response
  3. PROPFIND response for user address set
  4. PUT request and response
  5. REPORT response

After properly sending messages, the calendar could initialized with items. Added responses and configurations in js objects, so server configurations can be changed easily.

github link for the test :
https://github.com/Malintha/gsoc-14/blob/master/test_fakerServer.js

After the initialization of the calendar properly, I’m moving on with adding schedule-tag and if-schedule-tag-match test cases to the code.

Schedule-tag changes are evaluated on 2 scenarios.

  1. Changes to Organizer’s object resource
  2. Changes to Attendee’s object resource

For organizer’s changes, schedule-tag may or may not be changed on the type of the change.
[Refer http://tools.ietf.org/html/rfc6638#section-3.2.10]

  1. If the organizer changes the ical object with a PUT request, schedule-tag of the organizer’s object should be changed and changes should be merged with attendee objects[1].
  2. Schedule tag will not be changed as a result of processing an attendee PARTSTAT change response.

For attendee object resources there are three cases affect to the schedule tag.

  1. [1]Schedule-tag should be changed after processing updates from the organizer which contains updates more than other attendees’ PARTSTAT changes.
  2. Schedule-tag should not be changed after processing updates from organizer which only contains PARTSTAT changes of other attendees[2].
  3. Schedule-tag should be changed after processing HTTP requests like PUT,COPY,MOVE from the attendee himself.

It took me sometime to understand the scenario from the server perspective. [2]I’m not sure is it jumping into a prejudgement by understanding that, according to 2nd case about attendee’s schedule-tag change, server leaves attendees outdated by not letting clients updated after processing changes from other attendees.

On the implementation, I’m implementing the organizer scenario first with changes to the object resource. Implementation goes like following.

  1. change the existing calendar item(date/time, summary)
  2. send the changed item to the calendar by cal.modifyItem(Item,oldItem,listener)
  3. merge the change with attendee object resource and update the schedule-tag after PUT request
  4. reply to the multistatus message with new schedule-tag
  5. assert client’s item and servers item to make sure both are similar

Cheers!

Mid Term Passed – Game on its course

Mid term evaluation results were posted on yesterday by Google and I have passed! Cheers! :)

Last week was kind of a milestone of fakeserver test journey. I could establish the communication between the calendar and the test successfully and transmit upto multiget request between the calendar and the test.
The test up-to now.
Currently the test has 2 path handlers.

  • A calDav calendar was created on http://localhost/calendar and a handler was created to manage requests coming to that path.
  • A path handler to serve requests to the event url which is http://localhost/calendar/event.ics

Sequence of messages flow between the test and the calendar.

  1. Test sends addItem() message with PUT method to the calendar.
  2. Calendar sends back a multiget message to retrieve calendar data with REPORT method.
  3. Test serves the multiget request with Schedule-tag and, etag, href address and the actual calendar data. (Now, this is where I currently stopped. multigetRequestHandler’s SAXparser throws a fatal error at the response which is believed to be due to an error with wellformness of the response.)

Here onwards things have to be dealt with cautious. After move onto add the item successfully to the calendar, operations which the schedule-tag involved in have to be implemented. For that, schedule-tag needs to be calculated according to the RFC standards. Already created the method for schedule-tag manipulation, but definitely needs more situations handled when things move on.

I use wireshark to capture the packets which are being transferred between two endpoints. This is a screen shot of the first win. :)

Firstwin

Another todo : Needs to move handler functions to separate .sjs files for more clarity. But at the moment I thought to move on leaving them in the test file.

Fake Server Test : contd

It has some serious operations! Since the server needs to provide functionality to Create/Modify/Delete operations on requests, the logic behind the RFC standard operations needs to be embedded in the server backend.

During the test, server is created on localhost:50001 (I used the particular port, so it is easy to trace packets when comes to debugging) and requests are sent to specific URIs.

ie: create operation may send a request to <BASE_URL>/create/event.ics with PUT header. Also, If-None-Match : * header specified. We assume there are no conflicts with other files and etags are manipulated perfectly. Since the server is not a real file container, instead of creating temp files at the runtime (for new resources) and bind to the URL, an ics file will be registered to a URL and will be served by <BASE_URL>/create/event.ics url.

Once the file is asserted existing, the server will return 201 status code, etag and the schedule-tag. Calendar needs to keep them stored.

I spent the last week getting familiar with httpd.js, NSIHTTP* api and other test, util modules. I could create the basic handler functions and listeners needed to manipulate requests/responses. I found using wireshark to capture packets travel between two parties very helpful to move forward with the test.

My next target is to handle various requests on various URLs and treat them accordingly.

Unit Testing

It is time for testing. We decided that the test needs to be a unit test which can more generally surpress  calendar operations, with sequence of message injections.

That way we can make sure calendar operations are taking place according to RFCs and reacts to predefined behaviour according to given responses. XPCshell based unit test will provide preloaded response data imitating various CalDav server behaviours.

Unit test will basically create a temporary HTTP server on localhost using httpd.js. Calendar will send sync requests to the server and server will respond to them with different response data. Calendar reactions will be asserted against expected values.

As we decided, the test needs to be modular enough to mimic behaviour of different CalDav servers and changing the response data easily. The most challenging thing would be bringing the modularity to the test.

The test will contain :

  • Send initial PROPFIND request
  • Send multistatus requestto get hrefs of events
  • Send multiget request
  • Inject calendar data
  • Assert for schedule-tag and etag values

Along with If-schedule-tag-match

It was another patch for schedule-tag with support to If-Schedule-Tag-Match property. I made Schedule-Tag property implementation  more pluggable, so it makes the isScheduleTagSupport attribute true at startElement() of SAX parsing if the server returns Schedule-Tag value and then proceed.

To support If-Schedule-Tag-Match, it was needed to change the If-Match request headers in DELETE, MODIFY operations in to If-Schedule-Tag-Match. But possibly still some servers don’t support processing Schedule-Tag. Therefore I added lines to check if the response contains the schedule-tag before setting the request header. If not, it continues with the usual Etag comparison.

If the schedule-tag value belongs to the response is in the mItemInfoCache, there is no possibility that the server not support the header.

Still a lot of testing has to be done on the implementation, though it looks simple, it involves in all critical operations.

I hope to work on adding few more functionalities to the mozmill test for the imip-bar and more testing on schedule-tag implementation on next couple of days.

Cheers!

Multiget, multiget everywhere!

I spent last days going through sequence of handler functions and understanding at what points I should query for schedule-tag from the server.

There are 3 handlers in the calDavRequestHandlers.js

etags handler
webdavSync Handler
multigetSync Handler

Each handler parses the retrieved xml REST message with nsiSaxParser. At the end of parsing, [when reached to endDocument() function], it calls doMultiget() with a set of items that are identified need fetching.

If an item detected which was deleted, newly created or updated from etags it will be added to the list, itemsNeedFetching{}.

Items will be fetched in doMultiget().

So I decided adding the query, <C: schedule-tag> within multiget would do the thing. (WebDav sync doesn’t return a schdule-tag anyway)

Then the schedule-tag will be added to the list, mItemInfoCache with respected UID with  property name “scheduleTag” (I removed the hyphen in middle of the Local name). This may cause to add a null value for the mItemInfoCache[item.id].scheduleTag and I will manage it in If-Schedule-tag-Match property implementation.

I think this will pretty much cover the retrieval of schedule-tag property.

I found another regression in the code, which spoof the OPTIONS request when dealing with Google calendar.  I found that currently with CalDav v2, Google supports OPTIONS method. So there won’t be any need to spoof the method.  I’ll put this in my todo list as well.

My next step is implementing if-schedule-tag-match property.

New Headers : Schedule-Tag

Schedule-Tag is a new response header defined in RFC 6638 which is somewhat similar to ETag property of resources. The rationa behind the Schedule-Tag, is handing over the responsibility of merging changed object resource properties to the server.
With the current implementation, after every scheduling operation of invited calendar user, (Attendee changes the PARTSTAT value) causes to change the state of the object resource, which simply changes the ETag value and cause to occur inconsequential changes in the calendar user’s data due to some operations which possibly not affects to his/her PARTSTAT value.
Unlike the ETag value, Schedule-Tag only changed after the organizer made a change to the object resource .
Once a client tries a PUT request on an object resource, s/he should specify the last known Schedule-Tag of the object which is being asked with if-schedule-tag-match header which will be implemented in later stages. In case that the object resource has been modified with inconsequential changes, they have to be merged to the attendee object resource within the server and the new value has to be updated. Otherwise server must send a 412 (pre condition violated) status code in the response header.
Bug for the feature is here.
Changes of schedule-tag property in different situations.
I noticed that SOGo server, which I have used to work on has not yet support the schedule-tag property as it doesn’t return a value for schedule-tag value in the request header. Therefore, all current testing is being done with Google calendar.

Mozmill test for IMIP bar

There was a change in the planned routine in last week.  Mohit and I decided to poke into mozmill testing of IMIP bar though it was scheduled to latter stages of the project.

Bug for the test is here.

I had to look into mozmill automated testing API in order to write the test. Mozmill contains with some modules shared between firefox,  thunderbird and specific util modules for lightning. Mozmill thunderbird helper classes provide access to control utilities in thunderbird  while calendar utility classes provide ability to manage with events.

The automated test needed to inject a mail containing an invitation to check whether the IMIP bar is loaded and buttons are displayed. Then accept the event and make sure it is added at the correct location in the calendar view.

Source for the test is here.

In order to attach the invitation (invite.ics) to the email, I had to use thunderbird’s attchment-helpers module along with some other modules from tb as well.

There were few hard cases. First imip bar didn’t showed up and the attachment was showed up in the body. Once going through related RFCs, could find that, it was a some theoretical matter of adding the attachment header as defined in RFC.

Again, it was automatically clicking on “Don’t send email notifications” in the Email notification sending dialog box which appears once it’s clicked on the accept invitation button. gMockPromptService module was pretty much helpful for that. gMockPromptService creates the dialog virtually and returns the value (we can change the return value) to the caller regardless of the context of the caller.

In debugging I found that test functions are not executed unless the function name isn’t starting with the prefix “test”. That was pretty much helpful for debugging.

Cheers!

Selection_0011


Screenshot-from-2014-05-14-2308073

Deepening in the code

spent last few days exploring the code base and setting up the environment to debug Thunderbird.  Used sogo server to configure the CalDav. As I have installed a 32bit version of ubuntu, had to switch on Intel’s virtuaization feature in order to use sogo’s 64bit zeg image.

A summary on some frequent files would have been a little useful.

calUtils.js

This contains utility methods to be used in Lightning operations. ie: cal.LOG(); for logging messages in the error console.  This is loaded in many of the source codes.

calIitipUtils.jsm

Most of the operations related to itip is contained in this file. i.e: determining the latest resource of a event file by examining the sequence number and the datetime stamp.

calProviderUtils.jsm

This contains function implementations for provider services. ie: SendHttprequest to send the queried XML message to the server.

calDavCalendar.js

contains all the methods to send the initiate the PROPFIND request to the server to, deleting a calendar resource object from the remote calendar. ie:

Sending the initial PROPFIND request to find  the remote calendar.

sending a PROPFIND request with a “allprop” element is described here.

Initial PROPFIND request:

<D:propfind xmlns:D=”DAV:” xmlns: CS=”http://calendarserver.org/ns/” xmlns:C=”urn:ietf:params:xml:ns:caldav”>
<D:prop>
<D:resourcetype/>
<D:owner/>
<D:current-user-principal/>
<D:supported-report-set/>
<C:supported-calendar-component-set/>
<CS:getctag/>
</D:prop>
</D:propfind>

Response from the server:

 <?xml version=”1.0″ encoding=”utf-8″?>
<D:multistatus xmlns:a=”urn:ietf:params:xml:ns:caldav” xmlns:b=”http://calendarserver.org/ns/” xmlns:D=”DAV:”>
<D:response>
<D:href>/SOGo/dav/sogo1/Calendar/personal/</D:href>
<D:propstat>
<D:status>HTTP/1.1 200 OK</D:status>
<D:prop>
<D:resourcetype>
<D:collection/><calendar xmlns=”urn:ietf:params:xml:ns:caldav”/>
<vevent-collection xmlns=”http://groupdav.org/”/>
<vtodo-collection xmlns=”http://groupdav.org/”/>
<schedule-outbox xmlns=”urn:ietf:params:xml:ns:caldav”/>
</D:resourcetype>
<D:owner xmlns:D=”DAV:”>
<D:href>/SOGo/dav/sogo1/</D:href>
</D:owner>
<D:current-user-principal xmlns:D=”DAV:”>
<D:href>/SOGo/dav/sogo1</D:href>
</D:current-user-principal>
<D:supported-report-set xmlns:n2=”urn:inverse:params:xml:ns:inverse-dav” xmlns:n3=”urn:ietf:params:xml:ns:carddav”           xmlns:D=”DAV:” xmlns:n1=”urn:ietf:params:xml:ns:caldav”>
<D:supported-report>
<D:report><n1:calendar-multiget/>
</D:report></D:supported-report>
<D:supported-report>
<D:report>
<n1:calendar-query/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<n2:acl-query/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<D:sync-collection/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report><D:expand-property/>
</D:report></D:supported-report>
<D:supported-report>
<D:report>
<n3:addressbook-query/>
</D:report>
</D:supported-report>
</D:supported-report-set>
<n1:supported-calendar-component-set xmlns:n1=”urn:ietf:params:xml:ns:caldav” xmlns:D=”DAV:”>
<n1:comp name=”VEVENT”/>
<n1:comp name=”VTODO”/>
</n1:supported-calendar-component-set>
<b:getctag>1399188135</b:getctag>
</D:prop></D:propstat>
</D:response>
</D:multistatus>

I’m trying to send a HTTPrequest to get a property by name. It is done by the “propname” element.

my request:

<D:propfind xmlns:D=”DAV:” xmlns: CS=”http://calendarserver.org/ns/” xmlns:C=”urn:ietf:params:xml:ns:caldav”>
<propname/>
</propfind>

As long as the property name is not mentioned, it will be considered as “allprop” request.

As the next step I hope to jump into the implementation of schedule-calendar-transp property.

Entry Points

Automatic scheduling of calendar collections was introduced by CalDav extension which was a protocol built upon WebDav. (There is CardDav protocol introduced parallel to CalDav for transferring of vcard information) CalDav was introduced by the IETF from RFC 6638 but Lightning’s implementation was done before the final version was introduced. Referring to the draft version 4.

I went through both versions to identify changes between them and a list was prepared. I explored the bugzilla and the calendar blog to find bugs related to CalDav properties and  found some of them that can be used to track the implementation.

I started with the bug used to implement the CalDav in Lightning. https://bugzilla.mozilla.org/attachment.cgi?id=334246&action=diff

Also, hope I will be working the code around this bug to implement proposed schedule-default-calendar-URL Dav property in Lightning.  https://bugzilla.mozilla.org/show_bug.cgi?id=351745

My next step is to setup the Sogo server for testing features. It seems to be a very much updated and well aligned with the RFC 6638.

Also, I will be using next few days to hack into the code base to get a better understanding about the implementation along with RFCs. New bugs for each feature in the proposal are planned to be posted in the Bugzilla to get feedback and guidance from the community.

Cheers!

Copyright © 2014. Powered by WordPress & Romangie Theme.