End of the SoC, but the passion remains the same

That was a great period to remember for a life time. Reading, Coding, again reading, refactoring, some annoying issues (things get really annoyed when you update the local repository during a bustage ;) but still there was a lot of things to learn), getting reviews from experts..!!

I really enjoyed the SoC and hoping to continue contributing to internet giant Mozilla. There are few knits I still have to fix on patches and will be working on them in future as well. I got to know that fake server was something important to calendar so I will be focusing more into it and then fixing other proposed issues which were left behind due to sudden intrusion of the fake server.

Variations from the proposed deliverables as I decided with Mohit:

  • Week 3: Patch for schedule-calendar-transp
    Dropped since it requires some UI related changes
  • Week 4 – Week 6: Patch for If-Schedule-Tag and If-Schedule-Tag-Match
    Patch is done, needs to be made reviewable & handle edge cases
  • Week 6- Week 7: Patch for schedule-default-calendar-url
    No need to implement in lightning since this is more of a property that will allow clients to manage calendar collections for one user. More suited to a dedicated client for a particular server.
  • Week 7- Week 8: Patch for min/max date-time
    That is something related to the server, and is not returned in a PROPFIND request.
  • Week 8- Week 9: Patch for max-attendees per instance
    UI Changes required
  • Week 10 – Week 12: Unit tests for schedule-tag
    On track, this evolved into a major chunk of the project, with the fake server implementation and its design to support overriding, a rough working patch is completed, a more robust implementation in the works.
  • Week 12- Week 14: Mozmill tests for invitation scenarios
    This has been completed. Just some minor cleanup required with the packaged test thing.
  • Week 14- Week 15: Patch for schedule-force-send
    Needs UI change became less of a priority.

On the way, fake server became something that took a long time since it was a RFC implementation rather than a modification.

Throughout the GSoC period I received a great mentoring from Mohit, so herewith I’d like to give my gratitude to him for all the time he allocated and being cool throughout the period. Also, to Philipp for all the tech reviews. Without them I’d never caught up the complex things in the codebase.

Fun Facts :

Well, If you are going to contribute to Mozilla TB or Lightning by any chance lol, you should probably know these two abbreviations. You can’t find them in Google anyway.

m-c and c-c : I drove into trouble several times by hearing to update m-c and c-c. I was thinking that is something related to changset headers and hashes. well at last got to know, they weren’t. Those were the two folders named “mozilla-central” and “comm-central” in the repository.

PS: I searched in google for the term “c-c and m-c revisioning” and got the result “Women’s Choices Within Market Constraints: Re-Visioning;)

I will update the blog with my future works related to Lightning as well.

Cheers!

Fake Server – New Structure

There were a structural change in the fake server test I had to do before I submitting my last patch. Now the fake server is not merely a test and it can be extended by child classes to create desired types of server classes to be tested. Also I added a random e/schedule tag generator for events and memory calendar meta data API to keep them in the server. Hopefully, This will pretty much cover basic requirements of a fake server and can test the schedule tag as we expected. Schedule-Tag test and fake server class will go as two separate deliverables and schedule tag test will be based on the fake server class.

Fake server test : contd

The error raised in the PROPFIND block could fixed with the help of Philipp. It was due to wrong sequence of initialization messages to the calendar. To initialize a CalDav calendar properly, a sequence of asynchronous messages need to be sent.

  1. Initial PROPFIND response
  2. OPTIONS response
  3. PROPFIND response for user address set
  4. PUT request and response
  5. REPORT response

After properly sending messages, the calendar could initialized with items. Added responses and configurations in js objects, so server configurations can be changed easily.

github link for the test :
https://github.com/Malintha/gsoc-14/blob/master/test_fakerServer.js

After the initialization of the calendar properly, I’m moving on with adding schedule-tag and if-schedule-tag-match test cases to the code.

Schedule-tag changes are evaluated on 2 scenarios.

  1. Changes to Organizer’s object resource
  2. Changes to Attendee’s object resource

For organizer’s changes, schedule-tag may or may not be changed on the type of the change.
[Refer http://tools.ietf.org/html/rfc6638#section-3.2.10]

  1. If the organizer changes the ical object with a PUT request, schedule-tag of the organizer’s object should be changed and changes should be merged with attendee objects[1].
  2. Schedule tag will not be changed as a result of processing an attendee PARTSTAT change response.

For attendee object resources there are three cases affect to the schedule tag.

  1. [1]Schedule-tag should be changed after processing updates from the organizer which contains updates more than other attendees’ PARTSTAT changes.
  2. Schedule-tag should not be changed after processing updates from organizer which only contains PARTSTAT changes of other attendees[2].
  3. Schedule-tag should be changed after processing HTTP requests like PUT,COPY,MOVE from the attendee himself.

It took me sometime to understand the scenario from the server perspective. [2]I’m not sure is it jumping into a prejudgement by understanding that, according to 2nd case about attendee’s schedule-tag change, server leaves attendees outdated by not letting clients updated after processing changes from other attendees.

On the implementation, I’m implementing the organizer scenario first with changes to the object resource. Implementation goes like following.

  1. change the existing calendar item(date/time, summary)
  2. send the changed item to the calendar by cal.modifyItem(Item,oldItem,listener)
  3. merge the change with attendee object resource and update the schedule-tag after PUT request
  4. reply to the multistatus message with new schedule-tag
  5. assert client’s item and servers item to make sure both are similar

Cheers!

Mid Term Passed – Game on its course

Mid term evaluation results were posted on yesterday by Google and I have passed! Cheers! :)

Last week was kind of a milestone of fakeserver test journey. I could establish the communication between the calendar and the test successfully and transmit upto multiget request between the calendar and the test.
The test up-to now.
Currently the test has 2 path handlers.

  • A calDav calendar was created on http://localhost/calendar and a handler was created to manage requests coming to that path.
  • A path handler to serve requests to the event url which is http://localhost/calendar/event.ics

Sequence of messages flow between the test and the calendar.

  1. Test sends addItem() message with PUT method to the calendar.
  2. Calendar sends back a multiget message to retrieve calendar data with REPORT method.
  3. Test serves the multiget request with Schedule-tag and, etag, href address and the actual calendar data. (Now, this is where I currently stopped. multigetRequestHandler’s SAXparser throws a fatal error at the response which is believed to be due to an error with wellformness of the response.)

Here onwards things have to be dealt with cautious. After move onto add the item successfully to the calendar, operations which the schedule-tag involved in have to be implemented. For that, schedule-tag needs to be calculated according to the RFC standards. Already created the method for schedule-tag manipulation, but definitely needs more situations handled when things move on.

I use wireshark to capture the packets which are being transferred between two endpoints. This is a screen shot of the first win. :)

Firstwin

Another todo : Needs to move handler functions to separate .sjs files for more clarity. But at the moment I thought to move on leaving them in the test file.

Fake Server Test : contd

It has some serious operations! Since the server needs to provide functionality to Create/Modify/Delete operations on requests, the logic behind the RFC standard operations needs to be embedded in the server backend.

During the test, server is created on localhost:50001 (I used the particular port, so it is easy to trace packets when comes to debugging) and requests are sent to specific URIs.

ie: create operation may send a request to <BASE_URL>/create/event.ics with PUT header. Also, If-None-Match : * header specified. We assume there are no conflicts with other files and etags are manipulated perfectly. Since the server is not a real file container, instead of creating temp files at the runtime (for new resources) and bind to the URL, an ics file will be registered to a URL and will be served by <BASE_URL>/create/event.ics url.

Once the file is asserted existing, the server will return 201 status code, etag and the schedule-tag. Calendar needs to keep them stored.

I spent the last week getting familiar with httpd.js, NSIHTTP* api and other test, util modules. I could create the basic handler functions and listeners needed to manipulate requests/responses. I found using wireshark to capture packets travel between two parties very helpful to move forward with the test.

My next target is to handle various requests on various URLs and treat them accordingly.

Unit Testing

It is time for testing. We decided that the test needs to be a unit test which can more generally surpress  calendar operations, with sequence of message injections.

That way we can make sure calendar operations are taking place according to RFCs and reacts to predefined behaviour according to given responses. XPCshell based unit test will provide preloaded response data imitating various CalDav server behaviours.

Unit test will basically create a temporary HTTP server on localhost using httpd.js. Calendar will send sync requests to the server and server will respond to them with different response data. Calendar reactions will be asserted against expected values.

As we decided, the test needs to be modular enough to mimic behaviour of different CalDav servers and changing the response data easily. The most challenging thing would be bringing the modularity to the test.

The test will contain :

  • Send initial PROPFIND request
  • Send multistatus requestto get hrefs of events
  • Send multiget request
  • Inject calendar data
  • Assert for schedule-tag and etag values

Along with If-schedule-tag-match

It was another patch for schedule-tag with support to If-Schedule-Tag-Match property. I made Schedule-Tag property implementation  more pluggable, so it makes the isScheduleTagSupport attribute true at startElement() of SAX parsing if the server returns Schedule-Tag value and then proceed.

To support If-Schedule-Tag-Match, it was needed to change the If-Match request headers in DELETE, MODIFY operations in to If-Schedule-Tag-Match. But possibly still some servers don’t support processing Schedule-Tag. Therefore I added lines to check if the response contains the schedule-tag before setting the request header. If not, it continues with the usual Etag comparison.

If the schedule-tag value belongs to the response is in the mItemInfoCache, there is no possibility that the server not support the header.

Still a lot of testing has to be done on the implementation, though it looks simple, it involves in all critical operations.

I hope to work on adding few more functionalities to the mozmill test for the imip-bar and more testing on schedule-tag implementation on next couple of days.

Cheers!

Multiget, multiget everywhere!

I spent last days going through sequence of handler functions and understanding at what points I should query for schedule-tag from the server.

There are 3 handlers in the calDavRequestHandlers.js

etags handler
webdavSync Handler
multigetSync Handler

Each handler parses the retrieved xml REST message with nsiSaxParser. At the end of parsing, [when reached to endDocument() function], it calls doMultiget() with a set of items that are identified need fetching.

If an item detected which was deleted, newly created or updated from etags it will be added to the list, itemsNeedFetching{}.

Items will be fetched in doMultiget().

So I decided adding the query, <C: schedule-tag> within multiget would do the thing. (WebDav sync doesn’t return a schdule-tag anyway)

Then the schedule-tag will be added to the list, mItemInfoCache with respected UID with  property name “scheduleTag” (I removed the hyphen in middle of the Local name). This may cause to add a null value for the mItemInfoCache[item.id].scheduleTag and I will manage it in If-Schedule-tag-Match property implementation.

I think this will pretty much cover the retrieval of schedule-tag property.

I found another regression in the code, which spoof the OPTIONS request when dealing with Google calendar.  I found that currently with CalDav v2, Google supports OPTIONS method. So there won’t be any need to spoof the method.  I’ll put this in my todo list as well.

My next step is implementing if-schedule-tag-match property.

New Headers : Schedule-Tag

Schedule-Tag is a new response header defined in RFC 6638 which is somewhat similar to ETag property of resources. The rationa behind the Schedule-Tag, is handing over the responsibility of merging changed object resource properties to the server.
With the current implementation, after every scheduling operation of invited calendar user, (Attendee changes the PARTSTAT value) causes to change the state of the object resource, which simply changes the ETag value and cause to occur inconsequential changes in the calendar user’s data due to some operations which possibly not affects to his/her PARTSTAT value.
Unlike the ETag value, Schedule-Tag only changed after the organizer made a change to the object resource .
Once a client tries a PUT request on an object resource, s/he should specify the last known Schedule-Tag of the object which is being asked with if-schedule-tag-match header which will be implemented in later stages. In case that the object resource has been modified with inconsequential changes, they have to be merged to the attendee object resource within the server and the new value has to be updated. Otherwise server must send a 412 (pre condition violated) status code in the response header.
Bug for the feature is here.
Changes of schedule-tag property in different situations.
I noticed that SOGo server, which I have used to work on has not yet support the schedule-tag property as it doesn’t return a value for schedule-tag value in the request header. Therefore, all current testing is being done with Google calendar.

Mozmill test for IMIP bar

There was a change in the planned routine in last week.  Mohit and I decided to poke into mozmill testing of IMIP bar though it was scheduled to latter stages of the project.

Bug for the test is here.

I had to look into mozmill automated testing API in order to write the test. Mozmill contains with some modules shared between firefox,  thunderbird and specific util modules for lightning. Mozmill thunderbird helper classes provide access to control utilities in thunderbird  while calendar utility classes provide ability to manage with events.

The automated test needed to inject a mail containing an invitation to check whether the IMIP bar is loaded and buttons are displayed. Then accept the event and make sure it is added at the correct location in the calendar view.

Source for the test is here.

In order to attach the invitation (invite.ics) to the email, I had to use thunderbird’s attchment-helpers module along with some other modules from tb as well.

There were few hard cases. First imip bar didn’t showed up and the attachment was showed up in the body. Once going through related RFCs, could find that, it was a some theoretical matter of adding the attachment header as defined in RFC.

Again, it was automatically clicking on “Don’t send email notifications” in the Email notification sending dialog box which appears once it’s clicked on the accept invitation button. gMockPromptService module was pretty much helpful for that. gMockPromptService creates the dialog virtually and returns the value (we can change the return value) to the caller regardless of the context of the caller.

In debugging I found that test functions are not executed unless the function name isn’t starting with the prefix “test”. That was pretty much helpful for debugging.

Cheers!

Selection_0011


Screenshot-from-2014-05-14-2308073