LinuxCon + CloudOpen + ELC Europe 2014 and Linux Plumbers Conference 2014

Conferences are a way to know more about the projects and community and to network with some awesome people. And if it is LinuxCon Europe in conjunction with the Linux Plumbers then definitely it has to be an awesome awesome week!

About the conference

LinuxCon Europe is the place to learn from the best and the brightest, delivering content from the leading maintainers, developers and project leads in the Linux community and from around the world. Yes, it is an event where you can find all levels of technical talent gathered together under one roof for education, collaboration and problem-solving to further the Linux platform.

How I happened to be there?

I had been an OPW intern in the May – August, 2014 round and we had a Kernel Internship Report session organized by our coordinator, Sarah at the LinuxCon. Also, I had a talk about Coccinelle at the Linux Plumbers as a part of the developer tools microconference. So, I had a speaker ticket for two conferences. So, I got ready to fly to Dusseldorf – Germany, a beautiful city where I would get the opportunity to meet some awesome people and learn some great stuff.

How was the experience?

The keynotes sessions took place in a giant conference hall that accommodated about 1200 people. We had some experienced people talk about various open source initiatives like OpenMRS, how Amazon works with OpenSource projects etc. Jim Zemlin talked about the present state of Linux which was quite an interesting talk. There were talks about how huge communities are built and more technical subjects like wireless and the cloud. They were quite informative and broadened the scope of my thoughts. Then on the last day there was a discussion by the Linux creator Linus Torvalds with Intel’s Chief Linux and Open Source Technologies, Dirk Hohndel on the latest technical advancements in the kernel. It was the most awesome session with some brilliant and amusing answers by Linus.

I attended various talk which involved various subjects like SSH security, server optimization, MySQL availability. There was a session name “The cloud in 10000 words or 10 pictures” . It was the most awesome presentation I have ever been to not only because it is a topic of interest but because of the simplicity with which the speaker presented the entire matter. During the plumbers, I attended a tutorial on strace and learnt using various options for system call tracing.

I assisted Julia, my mentor from INRIA in a 2-hour long tutorial on Coccinelle and helped answer queries of the various attendees.

What I learnt?

I got an overview and some insights in various technical subjects such as the cloud, MySQL, system calls etc. I am now more aware of how the community was built, how open source projects are hatched and what it takes to manage such huge initiatives. It is sheer dedication and perseverance with a lot of patience that makes such projects successful.

I met a number of people from various open source and Linux projects like Yocto, elastic search, intel, enea, oracle etc. and learnt how some open-source projects are tweaked to serve specific applications.

I got to meet Greg, the most most patient kernel maintainer:

greg

I met Sarah, our coordinator who helped us feel at home by introducing us to the various community members. I met Julia, my awesome mentor who helped me throughout the internship and otherwise to help achieve my goals. We held a tutorial on Coccinelle and I presented Coccinelle in the developer tools micro-conference organized by her. My talks and help were appreciated by people around and it was very encouraging to know that developers are benefited by my work and experience.

We had some interesting events like the VIP dinner where all the speakers were there, then the Women in Open Source Networking Luncheon where all female community members met and the best one – an OPW dinner where all mentors, Sarah and the interns had food and some chat!

Dinner

Meeting people you have known through mailing lists and IRC was indeed wonderful. Talking to various kernel maintainers and developers about some prospective tasks and projects gave me new areas to work on to help develop the kernel.

So, it was one of the most enriching and amazing experience of my life. I would like to thank everyone there and the Linux Foundation for having such a huge event with a lot of fun and flair!!

Will keep you updated on more conferences and events to follow.. so stay tuned in!

Advertisements

All good things come to an end!

So, my OPW internship with the Linux kernel ends today. It has been an amazing journey throughout with the support of the awesome community and mentors. I would like to share a brief summary of my work, what I learnt and leave future novice contributors with a few suggestions.

My journey with the Linux Kernel began about 10 months ago. At college, I had been using Linux from the start and then we had a operating systems course. So, while learning the course and working on the assignments which involved the complete implementation of grub and the command-line, it came to my mind one day, why not look into the code of one of the greatest operating systems, Linux, that had been my primary OS for a long time now. Getting the linux repository turned out to be much simpler than I had thought. So, I began reading through the code, trying to understand and connect the dots, obviously I needed guidance. So, I did some research and got introduced to the awesome opensource Linux community through IRC and the mailing-lists and henceforth began the amazing journey of development with Linux kernel.

Reading through the code, I saw some inconsistencies in coding style and could conclude one of them must be the right way, but which. Then, the documentations came to the rescue. Linux has a short concise documents for everything like for the Coding Styles which is followed religiously across the kernel. There is a tool called checkpatch that serves just to enforce proper coding style in patches being created. So, I started off with small code cleanups, which were welcomed by the community and served a great encouragement.

I got interested in the endian-ness errors that have unfortunately found there way into the kernel and a tool called sparse has been developed to detect such errors. I ran the tool, analyzed the results(be careful there may be false positives) and sent-in patches to ensure the code does not crash on architectures with different endianness.

But as I got involved more, I found some errors that were common across many places and were quite specific. I happened to see some patches in the kernel that used Coccinelle(something that was entirely alien some months back :P) and realized that the coccinelle scripts could be run on any other file also and it would solve the same bug if it existed there. I came up with some simple Coccinelle scripts to detect and transform common bugs in the kernel. Then during the application period Julia had put up some challenge problems, that caught my attention and were very interesting to solve. The project Coccinelle went on to become my OPW internship project and my journey has been traced in the blog below.

Some major tasks done were:

1) Wrote Coccinelle scripts that are now in the scripts directory of kernel, helping developers to look for bugs to solve or detect bugs in the code being added.

2) I wrote a couple of managed interfaces for memory allocation so that the developers need not worry about explicit freeing of memory to prevent memory leaks.

3) Sent-in numerous patches solving bugs detected by the Coccinelle scripts and some others that I found along the way.

Major learnings:

1) I have a good understanding of how drivers are managed by the kernel, what happens when they are connected or disconnected, how failures are handled and so on.

2) I have developed insights to detect cases that can be solved by Coccinelle, as I am now aware of the various facilities provided by the scripting language, its capabilities and limitations.

3) I have gained a broad level understanding of how such a huge project runs smoothly with thousands of community members across the globe, helping each other contribute and help maintain the kernel.

Few suggestions that were important points I learnt from the community:

1) Before making a general change, maybe by looking at the previous patches etc., understand the code. There might be some peculiarity hidden in the similar looking code. When I say understand the code, I mean be sure of the impact of the change, why it is being made and what other ripple effects it can cause :).

2) Rather than guessing things that you only have a small clue of, go ahead and ask the developers, most of them are very active and would be happy to answer your questions.

3) While writing be specific, concise and clear. The community is huge with each member having other commitments as well, so it is good if he has to spend less time understanding your query.

I have through this journey learnt that apart from great coding, Opensource is about collaborative learning and innovating ..Its about the thrill of solving your first bug, making friends, improving your skills, experiencing lows.. but cutting through them and rediscovering your highs. Its about joining a community (and a family) and never wanting to leave!

I would like to express my gratitude to everyone in the Linux community. Special thanks to Julia for being the most awesome mentor ever, always encouraging me, helping me improve, allowing me to experiment and learn a lot :).

I won’t say a goodbye as I am not leaving :p (Yes! I plan to stick around for a long, long time !)

Signing off!

-Himangi

Scripts, patches and fun with them

These two weeks have been great fun, working with some Coccinelle scripts and sending in patches.

So lets look at a few scripts. These may look a bit too structured for a blog but wanted to keep it understandable thats why its so :P.

Aim : This script removes null test on values that cannot be null as they are assigned the value that has the form &a->b (add the offset of b to the address of a where a is a valid pointer) or are assigned using get_drvdata functions.
Purpose : This helped in removal of good amount of code within the NULL test block in some instances and also reduces unnecessary checks.
Coccinelle script :
@r@
expression e,f;
identifier g,y;
statement S1,S2;
position p1,p2;
identifier accessor =~ “get_drvdata”;
@@

e =@p1 &f->g
<+… \(f->y\|accessor(f)\)
…+>
if (e@p2 != NULL || …)
S1 else S2

@bad@
expression e,e2;
position r.p1,r.p2;
@@

… when != e =@p1 e2
e@p2

@depends on !bad@
expression e,e1,e2;
position r.p1,r.p2;
@@

* e =@p1 e2
… when != e = e1
when != &e
* e@p2

Aim: This patch removes magicnumbers and replaces them by existing constants:
Purpose: This will encourage use of the defined constants and help in better readability of the code for developers.
Coccinelle script:
@r@
type T;
T E;
identifier fld;
identifier c;
@@

E->fld & c

@s@
constant C;
identifier r.c;
@@

#define c C

@@
r.T E;
identifier r.fld;
identifier r.c;
constant s.C;
@@

E->fld &
– C
+ c

Aim: This patch eliminates NULL test after alloc_bootmem functions as they never return a NULL.
Purpose: Unnecessary code within NULL test is done away with as NULL values or ERR is never returned by alloc_bootmem functions. Also, useless checks are no longer performed.
Coccinelle Script:
@@
expression E;
statement S;
@@

E = \(alloc_bootmem\|alloc_bootmem_low\|alloc_bootmem_pages\|alloc_bootmem_low_pages\)(…)
… when != E
(
– BUG_ON (E == NULL);
|
– if (E == NULL) S
)

@@
expression E,E1;
@@

E = \(alloc_bootmem\|alloc_bootmem_low\|alloc_bootmem_pages\|alloc_bootmem_low_pages\)(…)
… when != E
– memset(E,0,E1);

Aim: Free memory allocated using kmem_cache_zalloc using kmem_cache_free rather than kfree.
Purpose: This will ensure proper deallocation of the cached content by calling cache_from_obj to get the kmem_cache and then free it.
Coccinelle Script:
@@
expression x,E,c;
@@

x = \(kmem_cache_alloc\|kmem_cache_zalloc\|kmem_cache_alloc_node\)(c,…)
… when != x = E
when != &x
?-kfree(x)
+kmem_cache_free(c,x)

An important point, as no mistake is a small mistake!

I encountered an interesting case in Coccinelle due to a change in language semantics. Earlier, if a specified string was used with “declarer” it used to match only that declaration but now it declares a metavariable that can match any declarer. So, we should use “declarer name” instead.

This is matching any declarer (thing like DEFINE_MUTEX, etc) and hence I ended up with 435 cases and was perplexed that how can so many instances endup using spin_lock instead of a mutex_lock.
initial::
@def@
declarer DEFINE_MUTEX;
identifier m;
@@

DEFINE_MUTEX(m);

@@
identifier def.m;
@@

(
– spin_lock(&m)
+ mutex_lock(&m)
|
– spin_unlock(&m)
+ mutex_unlock(&m)
)

But, then I realized the problem and found the right cases where a lock is defined as a mutex lock but is used with spin_lock 🙂 .
final::
@def@
declarer name DEFINE_MUTEX;
identifier m;
@@

DEFINE_MUTEX(m);

@@
identifier def.m;
@@

(
– spin_lock(&m)
+ mutex_lock(&m)
|
– spin_unlock(&m)
+ mutex_unlock(&m)
)

I have added the managed interfaces for kasprintf and kvasprintf that help in allocating resource managed space and copying an existing formatted string into it. Here is the patch and yeah its accepted. My first two official managed interfaces 😀

Woof!! It was interesting to work with these patches and tackle the issues 🙂 But, there are many more to go. 😛
Will keep you posted about more…Bye

Interesting scripts and points encountered

These days I have been working with some Coccinelle semantic patches and fixing different bugs.
I have used a simple semantic patch to transform cases of if(expression) BUG() to BUG_ON(expression).
Here is the Coccinelle script for it:

@@
expression e;
@@
-if (e) BUG();
+BUG_ON(e);

I have worked with some patches that correct the argument to a memory allocation function when the allocated data is a double pointer and erroneously the elements are sizeof the struct instead of pointer. I have hardened the semantic patch with all four modes. Here is the basic patch for the transformation:

@disable sizeof_type_expr@
type T;
T **x;
@@

x =

<+…sizeof(
– T
+ *x
)…+>

I have also had some patches fixing bugs involving converting !x & y to !(x & y). The semantic patch used for it is:

@@ expression E1,E2; @@
(
!E1 & !E2
|
– !E1 & E2
+ !(E1 & E2)
)

I had added a managed interface devm_gpio_request_array which allocates multiple GPIOs in a single call in a managed manner but the function addition was already in queue. It was however fun coding up my first devm function :P.

Many unmanaged to managed interface conversion patches have been applied :). Also, I learnt an important point while introducing managed interfaces in the ASoC driver. Using the managed functions within the ASoC level probe functions doesn’t work as devm_ is only usable as part of the driver model binding and unbinding. All the resource allocation needs to be moved into the device level probe (which is a better thing anyway) before it is converted to devm. So, I set out to analyze the various types of probe functions to find which probe functions already use devm. A list of the various probe functions using devm functions and having possibilities of some conversions was made using Coccinelle. Also, I am looking at some uses of managed interfaces in the remove functions understanding the usage as it is a bit stange.

Thanks Julia and the awesome Linux community for being the best mentors and guides and letting me experiment and learn SOOO much 🙂

Scripts are wonderful

The past couple of weeks have been quite eventful with some interesting scripts being written and tested out. So, lets talk about them.

There are some cases where the values returned by memory allocation functions like kmalloc, kzalloc, kcalloc, kmem_cache_alloc, kmem_cache_zalloc, kmem_cache_alloc_node, kmalloc_node and kzalloc_node are type casted, which is unnecessary. On sending a coccinelle semantic patch making the change, I realised from the feedback that not only these functions, but most void * values need not be type casted.
Here is a part of the initial script:

@@
type T;
@@

– (T *)
(\(kmalloc\|kzalloc\|kcalloc\|kmem_cache_alloc\|kmem_cache_zalloc\|
kmem_cache_alloc_node\|kmalloc_node\|kzalloc_node\)(…))

I started working on making the script more generic. But there were many cases where the void * has to be casted to make it useful. These are listed with examples to make them clear, but this may not be comprehensive:

1. Arithmetic operations. eg: sch->padded = (char *) sch – (char *) p; [Code]

2. Structure dereference. eg: n = ((struct rt6_info *)v)->dst.rt6_next; [Code]

3. Indirection operation. eg: *((u8 *) ptr++) = type; [Code]

4. Array dereference. eg: key = (struct sadb_key*) ext_hdrs[SADB_EXT_KEY_ENCRYPT-1]; [Code]

5. Cases where the type cast is __user, __ioremap, __force etc. which are not data types but macros. eg:  memcpy(to, (__force void*)from, n); [Code]

6. Casting of NULL before dereferencing. eg: dinfo->count = cpu_to_le32(sizeof(((struct aac_get_config_status_resp *)NULL)->data));[Code]

7. Use of sizeof on the casted value. eg: target_host->max_cmd_len = sizeof ((struct srp_cmd *) (void *) 0L)->cdb;[Code]

Taking into consideration these cases, I decided to handle only a assignment case initially. Here, is the patch.

@r@
void* e;
identifier a;
type T;
position p;
@@

a = (T@p *)e

@script:python@
t << r.T;
pc << r.p;
@@
if “__” in t: //To deal with cases like __user
cocci.include_match(False)

@remove_cast@
expression e;
identifier a;
type T;
position r.p;
@@

a =
-(T@p *)
e

Now, I am working on extending the patch to generic cases like when arguments are passes to a function with typecasting etc. being careful not to generate false positives.

And yeah for those following the blog, some more devm_ patches have got into the kernel and more will be following. I’ll be updating the list below soon. Many more interesting scripts and the complete version will be updated soon.. so keep learning and have fun 🙂

Coccinelle format and IRQs

Hey! The last couple of weeks have been great. I have understood and played with the format of the Coccinelle semantic patches in the kernel so lets talk about it a bit. A Coccinelle script has 4 modes of running – context, report, org and patch. The context mode just marks out the changes that will be done without actually making the changes, so a developer can checkout if some changes are unnecessary or erroneous and ignore those files/ change the Coccinelle script. Then there’s a report mode which is the most frequently used mode and it just logs a custom message which has the lines and files with the warning or error. The org mode is rarely used and also lists down the cases in a TODO format with exact line number and column positions with the modification to be done. The patch mode is the default mode and it applies the changes to the files where it detects the problem.  Soon, there will be some semantic patches developed by me in the kernel for linux developers to check out for any warnings being introduced and also to fix existing issues!

There is another major thing that came to my notice while working with shifting resource allocation to managed interfaces. The call to request_irq allocates interrupt resources and enables the interrupt line and IRQ handling. From the point this call is made your handler function may be invoked. Since the handler function must clear any interrupt the board raises, it takes care both to initialize the hardware and to set up the interrupt handler in the right order. While using managed interfaces for allocating resources we skip the corresponding functions to free the resources like kfree, free_irq etc. as the resource is freed by the devres interface on driver detachment. But there may be a race condition in the frees. There may be instances where the hardware resources are freed before interrupt resources get freed. In such cases, the interrupt handler will try accessing resources that no longer exist leading to an error. So, we need to be careful when shifting resources allocated by request_irq/request_threaded_irq to the managed devm_request_irq function.

 Here are some details of the function:
int request_irq (unsigned int irq, irq_handler_t  handler, unsigned long  irqflags, const char * devname, void * dev_id);

irq      : Interrupt line to allocate
handler  : Function to be called when the IRQ occurs
irqflags : Interrupt type flags
devname  : An ascii name for the claiming device
dev_id     : A cookie passed back to the handler function

Devm Patches: Successes and Issues Encountered

Over the last 2 weeks I have learnt that there are many managed interface and of different types. I would like to share my experience with patches till now. I have tried to add links wherever possible for someone who is more interested in the discussions and patches.

Let me start with some good news. The following patches have been applied to the kernel :).

Now my journey has seen some issues as some of the devm functions are a bit complex and the corresponding free/unregister function need to be seen very carefully to decide whether to do away with them or not. Otherwise, we could cause the terrible memory leaks or dangerously introduce a use after resource deallocation which can immediately crash the kernel unless the associated structure has been reallocated.

  1.  drivers/input/misc/pcf50633-input.c : In this case, I had just changed one of the resources to be allocated using devm function which was earlier allocated with kzalloc(). Now, Dimitry pointed out that mixing managed and unmanaged resources can be dangerous as it is really hard to track which one should be freed and which will be freed automatically. On further investigation I found input_allocate_device() function has a corresponding devm_input_allocate_device() function but there were other functions  like kobject_get_path() which don’t have corresponding managed interface and hence the driver was skipped.
  2. drivers/tty/hvc/hvc_opal.c : In this there was a case where a return was being done without kfree() of resources preciously allocated using kzalloc. Shifting to devm_kzalloc would have changed the workflow, but it appeared more like a memory leak bug and asked a query to the maintainers. I got a reply that one of the entries can be populated with a pointer to a static instance (hvc_opal_boot_priv) for the early boot/debug console and shouldn’t be freed. Hence, the error path was allowed to leak.
  3.  drivers/input/serio/apbps2.c : In this case,  in the remove function there was a call to serio_unregister_port.  This function uses the kobject interface that is complicated and seems to ultimately free its argument. Looking at some other calls to serio_register_port and serio_unregister_port, it is found that they also use kzalloc and no kfree. So, in conclusion, it is a good idea to be suspicious if there is no kfree anywhere for the data.  It could of course be a memory leak, but it could also be that the data should not be kfreed.
  4. drivers/dma/iopadma.c : In the remove function, a list of iop_chans is being freed and then the device. So, apparently there is a list of iop_chans per device. The probe function also makes a list for the channels, and makes one iop_chan, and puts it in the list. But the fact that there is an iteration (list_for_each_entry_safe) suggests that there can be other elements of this list.  It is unclear as to how these elements are added. If they are added somewhere else with kzalloc, then we can’t devm the one added here, but not the other ones. I have asked  a query to the maintainers and am waiting for an answer.
  5. drivers/input/misc/sparcspkr.c : This could only be partially converted to use devm for resource allocation due to absence of a devm version of of_ioremap. A solution could be to introduce a devm version of the of_ioremap function as it is used commonly. A query has been asked to the maintainer and I am waiting for an answer.
  6. drivers/misc/eeprom/sunxi_sid.c : The resource entropy was being devmified but entropy is a buffer that is just allocated and freed within the normal execution of the function. The buffer is to be freed even when the probe function succeeds.  A devm buffer would only be freed if the probe function fails.  Otherwise, it will be kept around until the remove function succeeds.
  7. drivers/input/misc/mc13783-pwrbutton.c : The probe function used mc13xxx_irq_request which is not devm’d, and doing so is complicated due to the locks around the frees.  So, it was left to avoid mixing two styles.

The following patches though not yet accepted have been approved by the maintainers are expected to be accepted soon. Fingers crossed!!

Other patches submitted: