Monday, 26 September 2016

me et mentis morbum

Fig 1. What I aspire to be ...

This morning, I want to talk about mental illness ... and apologize.

The preceding sentence was rewritten 42 times, and contains the words from the first revision.

In some sense, it is difficult to use the words "mental illness" when we are talking about ourselves.

There is a kind of stigma attached to even thinking the words. Before you can think the words, you must first have the revelation that there is something wrong with the way you behave and or feel.

Some or all of the following things happen when I have to mix with other people in real life:
  • I will try to avoid it, completely.
  • I can't remember where I am, or where I am going, and appear and feel lost.
  • I struggle to recognize people, or remember names.
  • I don't move among humans like other humans do; I bump into them in closed spaces (for example, supermarkets) because I can't read their behaviour.
  • I become outwardly awkward, developing ticks (for example blinking), stuttering and repeating myself. 
  • I become inwardly slow, results in not being able to join in conversations properly:
    • Struggle to figure out if I agree or disagree with any statement.
    • Wonder if it is worth saying what I want to say.
    • Wonder if I might be stupid.
    • Fail to form a response before the conversation moved on.
    • Malform a response, with stuttering and repetition abound.
If I become conscious of any of the behaviours above, my physiology changes; I begin to radiate heat and sweat profusely. This always results in my ending it early and just leaving, often without announcing my departure.

There are a few people who make me feel somewhat at ease, just by their presence. People who are physically large (I am rather small), and attentive (go out of their way to patiently interact with me). I can't really explain this, but they seemingly have some immunity.

To compound the problems I have with social or stranger interactions, and possibly even as a result of it; I cannot remember, in any detail, for any appreciable amount of time, what has been said. With the occasional exception where the interaction resulted in a changed or new understanding of something I am interested in, I will normally have forgotten, everything, including the detail of where I was when anything happened, and usually how I got there (having driven, or ridden a motorcycle to the location), by the next day. I retain new technical information, or more generally ideas, usually, sometimes incorrectly. Of course I could be wrong about that, since I don't remember what I don't remember, but it feels that way. In the mundane, I might retain the price of a thing I'm interested in buying, or where to get it from, but may not remember where the information came from. I will eventually have some familiarity with a face, but it takes longer to remember the correct name, and doesn't always happen. I might never remember any particular location, and don't seem to posses the intuition I see others exercising when navigating the world, or even any particular building, or location of a certain size.

That memory problem might sound quite comical, even advantageous given our profession, but it results in a person that may give the first or persistent impression that they can't remember you, your face, your name, if you have children, or are married, or anything else about you. It creates a person that seems incapable of paying attention to other humans, that will deny having even met you, or having had that conversation which you really enjoyed, or I enjoyed.

For most of my professional life, I have been isolated from the real world. Communication with non familial humans has been mostly electronic, and while it may be a tempting explanation to invoke, it's not correct. These symptoms and behaviours pre-date my professional career, some existed as a pre-teen child.

Most of the time, I have a way above average output, not to boast, but I (usually) live to code ... my children have been late for school before (not by much) so I can finish writing or testing a bit of code - because driving to school while thinking about code is, of course, impossible. When I'm given a task, especially one that involves my doing stuff that hasn't been done before, or is exceptionally challenging, it consumes me until it is done. My enthusiasm delivers my above average output.

Twice now, the enthusiasm has been sucked out completely, and replaced by such a strong feeling of dread and worthlessness, that lethargy takes over and prohibits normal function in every area of life.

Many of you will be quick to label this as "burnout": When you can't find the enthusiasm you require to take your kids anywhere, or go to a school event for them, or go shopping for food, or make love to your wife, it is not burnout.

In our technical world, it's all too easy to dismiss behaviours that are symptomatic of mental illness as a product of our geeky-ness, and our working environments. I think this is one of the reasons why these behaviours have been accepted as normal (or normal-for-geeks) and then ignored, even by those closest to me.

The first time the enthusiasm disappeared, life pretty much fell apart, I had no idea what was happening.

I think, I am just recovering (with life intact) from the second time ...

It was extraordinarily difficult to approach my wife, and soul mate of more than a decade, and say out loud "I think I have a mental health problem". They are the words I used, I needed to be direct, unambiguous. By the time I had finished the sentence, I could see the understanding written on her face, the relief ... I'm lucky to have her.

Doctors appointments and diagnosis (acute social anxiety and depression) followed, and treatment is ongoing.

The antecedent factors that lead to these difficulties don't interest me very much,  and are shrouded in the blurriness of my memory whatever. Getting at the reasons seems all but impossible. What concerns me is how to live and function with these symptoms today, not the days or months from my childhood that may or may not have helped to form or exacerbate them.

Compared to the discomfort of telling my wife, telling my manager was easy. My manager has a beard, 
which obviously helps ... while my wife has no beard. In all seriousness, it was only easy because of the understanding nature of the response I received.

I'm not back to normal yet, there are good days and bad, but none of them are normal. I don't fully understand why this has happened, and tend to think it is not fully understandable in principle whatever.

I'm saying all of this in public for two reasons: I think I owe the community an explanation for my recent absence, an absence which may well effect your ability to do your job properly. Most importantly, I feel obliged to speak in order to normalize talking about it, and directly encourage anyone who thinks "that's me" when reading this, to get some help.

Even the internal dialogue that lead to the realization that I have to tell my wife anything at all, was, in some small way, liberating. Surely this helped to provide whatever I needed to first utter those words.

I can't apologize for being unwell, but I can and should apologize if I let you down without explanation; I'm sorry.

Wednesday, 13 April 2016

Breaking Badly


If you're not running PHP 7 already, you are either crazy, or else your unit tests rely on software that I wrote for PHP 5 ... uopz.

uopz is a runtime hacking extension of the runkit and scary stuff genre.

When I first wrote uopz, PHP 5 was almost in a state of equilibrium. There were minor changes effecting the kind of stuff those extensions do, but by the time 5.5 came the platform was more or less stable.

When PHP 7 came along, it was a major roadblock that we could not run our unit tests. Badoo and many other large projects also had the same problem.

While there were tickets open requesting support for PHP 7, I omitted to answer the tickets in favour of working on the code. I wrongly assumed, if you were waiting for updates you would be watching closely. I also omitted to tag any issues in any commits I was making simply because I'm bad at git.

Various tweets and tickets went unanswered ... sorry about that, but ... code and ... me.

For many months, you could compile uopz for PHP 7 and it would "work", but it was terribly unstable.

I have my fingers in many pies, maintain many things; uopz was not the only road block and not the most important thing to do. It did receive attention ... but I will admit, not enough attention.

More precisely, not enough of the right kind of attention. Every time I came to work on uopz I was focused on making it do exactly the same things it done before, in exactly the same way.

Some of the stuff uopz does is semi-ordinary, it even calls Zend API in a lot of cases. However, copying functions (in the bitwise+instruction by instruction sense), manipulating the global function table, and class function tables, is not ordinary; This is what makes uopz or runkit scary, and useful.

PHP 7 is vastly different to PHP 5 internally, the VM is a much more complicated place to try and get work done.

If we think about a year from now, or two years from now:
  • What happens when Zend has a JIT ? 
  • What happens if Opcache makes class entries immutable, and so shares them ?
  • What happens if Opcache makes function entries immutable and re-entrant, and so shares them ?
I do not know the answers to those questions, but they are good questions.

When these thoughts are communicated clearly, it might be obvious that uopz cannot work in the same way, and be stable, or forward compatible.

Function Mockery v5 

 

You do not need to delete, rename, or otherwise modify functions, or function tables.

The purpose of allowing you to delete, or rename, a function was to allow you to create another one in it's place.

The purpose of allowing you to create a new function in the place of the deleted function, was so that you could define new behaviour.

This is complicated by the fact that your new function may need to invoke the original function with certain parameters.

Almost certainly, at some time the original function will be explicitly restored, possibly after a group of tests (tearDownAfterClass perhaps).

Even if restoration is not performed explicitly, at request shutdown everything must be restored - you cannot leave a user function in the function table of an internal class, nor can you delete internal function entries earlier than the engine is expecting.

Simply deleting the function is not an option, you have to keep it, and ideally provide a way to copy it into a closure in userland.

That's a rather roundabout way of doing things, don't you think ?

Here's a better way:
function uopz_set_return(string class, string method, mixed value [, bool execute = false]) : bool;
function uopz_set_return(string function, mixed value [, bool execute = false]) : bool;

This new API does not modify the function table, instead it intercepts the execution of an existing function and allows you to set a return value.

The return value can be any variable, or a Closure to be executed in place of the original function, but still without modifying any function tables:
uopz_set_return('strlen', function(string $string) : int {
 return strlen($string) * 2;
}, true);

var_dump(strlen("four"));
The code above yields:
int(8)
In some cases, you do not want to modify the behaviour of the original function, but rather modify some state or perform some other action upon entry to a particular function:
uopz_set_hook('strlen', function(string $string) {
 echo "Expect: int(4)\n";
});

var_dump(strlen("four"));
The code above yields:
Expect: int(4)
int(4)
Hook and return closures are bound to the current scope at runtime:
class Foo {
 private $bar = true;

 public function qux() {
  return false;
 }
}

uopz_set_return(Foo::class, 'qux', function() {
 return $this->bar;
}, true);

$foo = new Foo();

var_dump($foo->qux());
The code above yields:
bool(true)
Setting hooks and returns should have most use cases covered, still there are times when you need to add a non-existent function:
function uopz_function_add(string class, string method, Closure handler [, int flags = ZEND_ACC_PUBLIC]);
function uopz_function_add(string function, Closure handler [, int flags]);
This new API allows you to do that, it is similar to uopz_function, but can only add functions, it will not replace functions.

I was reluctant to allow adding functions at all, it makes everything slower because it means we have to disable function entry caching. All of the uses we have left now are questionable, I suspect the same is always true.

The majority of the time, you were only adding a function because there wasn't a better way ... use the better way :)

Class Mockery v5

 

Allowing userland code to over ride opcode handlers was always bat shit crazy!

In PHP 5, to provide a mock class at test time, you had to overload ZEND_NEW and change the name of the class.

Not only is that crazy, it's bad.

If you happen to be running tests that create 1000 objects of a particular kind, but could use the same object, you are wasting the resources consumed by the creation of 999 objects. In test suites where we have tens of thousands of tests, this can have a dramatic effect.

In PHP 7 we have anonymous classes, this allows us to have a rather beautiful API:
interface IFoo {
 public function bar() : bool;
}

function consumer(IFoo $foo) : bool {
 return $foo->bar();
}

uopz_set_mock(Foo::class, new class implements IFoo {
 public function bar() : bool {
  return true;
 }
});

var_dump(consumer(new Foo()));
The code above yields
bool(true)
uopz_set_mock can also accept a class name as the second parameter, the following code will behave identically to the code above:
interface IFoo {
 public function bar() : bool;
}

function consumer(IFoo $foo) : bool {
 return $foo->bar();
}

class Mock implements IFoo {
 public function bar() : bool {
  return true;
 }
}

uopz_set_mock(Foo::class, Mock::class);

var_dump(consumer(new Foo())); 
Anonymous classes have superseded the role of uopz_compose, which used to allow composition of classes at runtime, in a rather awkward way. While uopz_compose was a nice toy, we are looking for stability, and forward compatibility, which are guaranteed if we are relying on language features.

Sorry 


I broke BC, and I feel bad about that ... my pain is eased by the thought that I provided a more stable, superior API to work with, that has a chance of being forward compatible with whatever Zend does next.

I also feel bad that I haven't had time to update the documentation for uopz yet; For now the README is the documentation.

If anyone feels like helping with documentation, that would be much appreciated.

Happy testing :)

Tuesday, 15 March 2016

Hacking PHP 7


Recently, I have taken part in some screen casts with my good friends at 3devs.

The subject of the screen casts are extension development for, and hacking PHP 7 (Part 1, Part 2).

Screen casting is a medium I haven't mastered, or had very much practice at.

While I'm trying to plan the content for the show, I can't help but be reminded of every lecturer and tutor that stood in front of me repeating facts, sometimes literally reading from a book, or their own notes.

As a rule of thumb, if someone says something to me, and my life does not depend on my retention of the information contained in the statement, I will immediately, and without prejudice to the speaker, forget what was said.

One of the first things we learn as children are our multiplication tables, and we first remember those which have a pattern, or some trick, to describe, or determine the sequence of numbers. We don't remember the sequence until we have uttered it at least a few hundred times. The same goes for the alphabet; We roughly remember a kind of melody, and get most of the letters in the second half of the alphabet wrong for quite some time.

As education progresses, perhaps for reasons of logistical expediency, the focus does shift from inspiring us, with the beauty of the rainbow, to learn how the rainbow works, to barking facts at us about the past wars and rulers of our respective country, and having us try to memorize various tables of information.

This sucks, it sucks so hard: Just when your mind becomes fully primed, and developed enough to really respond to inspiration, the inspiration is almost completely squashed from our education process.

Some may be lucky enough to have a better experience of education, but broadly speaking, until higher education anyway, this is how we were, and our children still are "taught".

It is the job of the teacher to inspire listeners to learn for themselves, it is not the job of a teacher to bark facts. I've really tried to convey that in the material we prepared.

Screen casting is a difficult medium however, so to accompany the screencast is this blog post.

Writing extensions is fun, but it's not as fun as hacking PHP. So, we're going to focus on hacking, we're going to imagine that we are introducing some new language feature, by RFC.

Without focusing on the RFC process itself, you need to know which are the relevant parts of PHP you need to change, in order to introduce new language features.

You also need to know how PHP 7 works, about each stage of turning text into Zend opcodes ...

In the Beginning: Lexing


When the interpreter is instructed to execute a PHP file, the first thing that happens is lexing, or if you prefer, lexical analysis.

The lexer, accepts a stream of characters as input, and emits a stream of tokens as output.

The input to the lexer is the characters of the code. The output is those sequences that the lexer recognizes, identified in a useful way.

The following function is illustrative of what a lexer, or lexical analysis does:
<?php
function lexer($bytes, ...) {
    switch ($bytes) {
        case substr($bytes, 0, 2) == "if":
            return TOKEN_IF;
    }
}
?>

A lexer doesn't actually work like that, but it's enough to understand what it does, you can read associated documentation, or source code to discover how it does it.

The lexer function itself is generated by software known as a "lexer generator", the one that PHP uses is named re2c.

The input file to the lexer generator is a file which contains a set of "rules" in a specific format.

I'll just take one illustrative excerpt from the input for the lexer generator:
<ST_IN_SCRIPTING>"if" {
    RETURN_TOKEN(T_IF);
}

This can be roughly translated to "if we are in a scripting state, and find the sequence 'if', return the identifier T_IF".

What "scripting state" means becomes clear when you remember that PHP used to be embedded in HTML: Not all of a file the interpreter reads is executable PHP code.

Consuming Tokens: Parsing


The output from lexical analysis now goes through the process of parsing, or if you prefer, syntactical analysis.

You can think of the parser as a syntax2structure function, which will take your stream of tokens produced by lexical analysis, and create some kind of representative data structure.

Once again, the parser function is generated, by software known as a "parser generator", the one that PHP uses is bison.

The input file to the parser generator, which is referred to as the "grammar", is another set of rules.

Here's my illustrative excerpt from the input for the parser generator:
    T_IF '(' expr ')' statement 
      { /* emit structure here */ }

This can be roughly translated as "if we find the syntax described, execute the code in braces".

Historically, pre PHP 7, parsing the code was the final step in compilation of your code: the parser emitted opcodes directly: we had one-pass compilation.

Today, in PHP 7, parsing the code is the penultimate step in compilation; The parser in PHP 7 emits an abstract syntax tree: we now have multiple-pass compilation.

In the End: Consuming AST


The final pass, is when we emit the target form, Zend opcodes.

Each node in the tree is passed in to a compiler function, depending on it's kind, which in turn passes it's child nodes in to their respective compiler functions, and so on ...

Here's my (abbreviated) illustrative excerpt:
void zend_compile_stmt(zend_ast *ast) /* {{{ */
{
 if (!ast) {
  return;
 }

 /* ... */

 switch (ast->kind) {
  /* ... */
  case ZEND_AST_IF:
   zend_compile_if(ast);
   break;
  /* ... */
  default:
  {
   znode result;
   zend_compile_expr(&result, ast);
   zend_do_free(&result);
  }
 }
}
/* }}} */

A new feature may not always require new abstract syntax, if this is the case, you can implement your RFC by using the AST, and Compiler API.

New abstract syntax will require a new compiler function, but may emit existing opcodes.

The VM


The virtual machine is the thing that executes opcodes; It's as much like a CPU, as opcodes are like instructions for a CPU.

For each opcode exists a handler, here is my illustrative excerpt:
ZEND_VM_HANDLER(43, ZEND_JMPZ, CONST|TMPVAR|CV, JMP_ADDR)
{
 USE_OPLINE
 zend_free_op free_op1;
 zval *val;

 val = GET_OP1_ZVAL_PTR_UNDEF(BP_VAR_R);
 
 if (Z_TYPE_INFO_P(val) == IS_TRUE) {
  ZEND_VM_SET_NEXT_OPCODE(opline + 1);
  ZEND_VM_CONTINUE();
 } else if (EXPECTED(Z_TYPE_INFO_P(val) <= IS_TRUE)) {
  if (OP1_TYPE == IS_CV && UNEXPECTED(Z_TYPE_INFO_P(val) == IS_UNDEF)) {
   SAVE_OPLINE();
   GET_OP1_UNDEF_CV(val, BP_VAR_R);
   ZEND_VM_JMP(OP_JMP_ADDR(opline, opline->op2));
  } else {
   ZEND_VM_SET_OPCODE(OP_JMP_ADDR(opline, opline->op2));
   ZEND_VM_CONTINUE();
  }
 }

 SAVE_OPLINE();
 if (i_zend_is_true(val)) {
  opline++;
 } else {
  opline = OP_JMP_ADDR(opline, opline->op2);
 }
 FREE_OP1();
 if (UNEXPECTED(EG(exception) != NULL)) {
  HANDLE_EXCEPTION();
 }
 ZEND_VM_JMP(opline);
}

New opcodes are going to need to new handlers, which you simply add to zend_vm_def.h, at the end of the list, using the next available opcode number.

The prototype for the macro ZEND_VM_HANDLER can be assumed:
    ZEND_VM_HANDLER(opnum, opname, op1_type, op2_type)

This is obviously not standard C code, this definition header file is used as input to generate the real handler functions for all the permutations of operand types used.

When changes are made to the header, you must regenerate the VM by executing zend_vm_gen.php (in the Zend folder).

The Hack


The first hack introduced in the screen cast was a hipster expression. The hipster expression looks like a function call (à la include_once). If the result of the expression passed to hipster is a string, the result is copied to the return value, or else an exception is thrown.

This is a completely useless feature, other than it's illustrative value.

Since we are introducing a new token "hipster", we must start by editing the lexer to include the following code:
<ST_IN_SCRIPTING>"hipster" {
 RETURN_TOKEN(T_HIPSTER);
} 

Care should be taken to make edits in sensible places in these generator input files.

The next thing to do, is making the parser aware of the new token.

Open up the parser generator input file and search for "%token T_INCLUDE", something like the following will be found:
%token T_INCLUDE    "include (T_INCLUDE)"

%tokens have the form %token ID FRIENDLY_NAME, we must add our new token:
%token T_HIPSTER "hipster (T_HIPSTER)"

Now search for "%left" (or scroll down), something like the following should be found:
%left T_INCLUDE T_INCLUDE_ONCE T_EVAL T_REQUIRE T_REQUIRE_ONCE

This section of the file is where operator associativity is set, and precedence implied in the order.

Since we are introducing a non-associative token we add the line:
%nonassoc T_HIPSTER

In an appropriate place, at the end of the first block of %nonassoc tokens, for example.

Now we must add a new rule to the parser in an appropriate place, since we are adding a new expression, we use the expr_without_variable rule.

In the example hack, we add:
 | T_HIPSTER '(' expr ')' { $$ = zend_ast_create(ZEND_AST_HIPSTER, $3); }

Underneath:
 | internal_functions_in_yacc { $$ = $1; }

Notice we are going to use a new kind of AST node, so the next thing to do is edit zend_ast.h to add our new ZEND_AST_HIPSTER to the enumerated _zend_ast_kind.

The enumerated kinds have a magical format that requires studying to determine where to add the new kind, we add it here:
        
    ZEND_AST_CONTINUE,
    ZEND_AST_HIPSTER,

Because our new node will have a single child, the expression.

The next thing to do is editing the switch in the zend_compile_expr compiler function, adding, in an appropriate place:
    case ZEND_AST_HIPSTER:
    zend_compile_hipster(result, ast);
    return;

Then we must define zend_compile_hipster:
void zend_compile_hipster(znode *result, zend_ast *ast) /* {{{ */
{
 zend_ast *expr_ast = ast->child[0];
 znode expr_node;

 zend_compile_expr(&expr_node, expr_ast);

 zend_emit_op(result, ZEND_HIPSTER, &expr_node, NULL);
} /* }}} */

At this point, the lexer, parser, and compiler are all aware of our new feature, but, we emit an opcode that does not exist.

So, we edit zend_vm_def.h to add our new opcode handler:
ZEND_VM_HANDLER(184, ZEND_HIPSTER, ANY, ANY) 
{
 USE_OPLINE
 zend_free_op free_op1;
 zval *op1;

 SAVE_OPLINE();
 op1 = GET_OP1_ZVAL_PTR_UNDEF(BP_VAR_R);

 if (Z_TYPE_P(op1) != IS_STRING) {
  zend_throw_exception_ex(NULL, 0,
   "hipster expects a string !");
  HANDLE_EXCEPTION();
 }

 ZVAL_COPY(EX_VAR(opline->result.var), op1);

 FREE_OP1();
 ZEND_VM_NEXT_OPCODE_CHECK_EXCEPTION();
}

Remembering to regenerate the vm after our change.

We're done, so execute make, and then marvel at:
sapi/cli/php -r "echo hipster('PHP 7 rocks') . PHP_EOL;"

Finally


The best way to learn about PHP internals is to dig in: If all of us got together in a room tomorrow, and did not leave until we had produced a volume of encyclopedic knowledge, the best way to learn would still be to dig in.

If everything were explained in exquisite detail here, you probably wouldn't have got to the end ... You may have to read this more than once, I'm sure you'll have to do a bunch of research besides ...

Hopefully, I've given you enough links, terms to research, and things to think about to inspire you to do just that ...

Happy hacking :)

Wednesday, 2 March 2016

Picking an Approach

Fig 1. Several Languages
I should hope that the majority of people reading this consider themselves polyglots.

A polyglot is a person able to speak in many languages; It's almost a requirement of programming that we should know more than one language.

Using the right language for the job is a worthy aspiration to have.

When you, or I, as a programmer are setting out to write a new product, or application, we should definitely consider our options: It's a matter of fact that no language is best for everything.

The right tool for the job makes sense as part of our approach to writing applications ...

Choosing Wisely


Fig 2. PHP Internals

When we approach the design of a language, I submit that it doesn't make sense to use the right tool for the job argument against having a feature.

We need to aim for something in our approach, but what should it be ?

Before I try to define that, we should probably admit that whatever our aim, we might miss by quite wide margins.

It's difficult to organize groups of individuals spread out across the world, when there isn't so much as a single person you could identify as "manager". Your team at work might be spread out across the world, but your teams at work are carefully managed.

We can choose an idyllic aim, even knowing that we will probably miss: Being anywhere close to an ideal, is better than having no aim at all!

I think, our aim should be to provide the best tool for the job.

It's true that no language is best at everything, it's also true that no language is good at everything. However, it has to be part of everybody's ideal that such a language should exist; One that is good at everything.

We will probably miss our target, PHP will never be best at everything, but in aiming for that, we have good chance, over a long period, many versions, of really having a language that is genuinely good at everything (that we care about at the time).

It still won't be the right tool for every job, there will be better languages, forever.

I want to see people stop advancing the right tool for the job as an argument, I want to see people accept foreign ideas more willingly if it creates a better tool for programming.

In the not too distant future, when things like async/await are suggested, don't be tempted to rebel against that because there are better tools for the job.

Don't shout "use go", even though we all know it's probably the best tool for that kind of work today.

Let us make PHP a realistic option for tomorrow, let us at least try to provide the best tool for the job ...

Back to code ...

Monday, 8 February 2016

Chasing the Dragon

Fig 1. Man chasing Dragon.
It's no secret that, I am an addict ...

I'm allowed to assume that if you are reading my blog, about programming, you too are an addict.

All of the time I am awake, I chase the ultimate high ... that high is ... making computers do stuff.

We probably both have a problem ...

I've always found "chasing the dragon" to be a beautiful metaphor for the pursuit of our aims, in spite of the damage it does to us.

Beautiful, right up until the moment the chaser catches the dragon, the dragon turns and kills the chaser.

I don't have the energy I did four, three, two, or even one year ago, do you !?

The pursuit of my aims is taking it's toll, days are shorter now, I can't concentrate for 16 hours at a time any more.

I'm acutely aware that I'm becoming less able.

Before the dragon turns and gets me, I have to change pace.

For some months now I have been trying to get out of the house at weekends (I work from home), stop working at a reasonable time, and spend time with the family and kids most evenings, rather than on code.

This is a difficult adjustment, days where I really feel I achieved something are vanishingly few, I'm assured this is all in my head by people I work with. That my perception may be skewed doesn't make the adjustment any easier.

If you're still reading, I like to think it's because my words are ringing in your ears: I could be describing a version of you, from not too far in the future ...

Maybe you have excellent foresight, and it won't come for another decade or two.

Mentoring as Substitute


Fig 2. Blob headed beings, whom I would like to meet.


I'm super fond of solving problems, and I don't care whose problem it is.

Within the PHP ecosystem, there is a sense in which all problems of a certain kind are my problems: Because there are so few of us (relative to the number affected) that are able (or willing) to solve the kinds of problems I'm talking about.

I love that. I can get my fingers in many pies, become familiar with software we all take for granted, even most other internal developers. It helps me to understand ... I don't know about you, but understanding doesn't come easy to me, I have to really work at it.

Electronically at least, I'm also super fond of talking; Ask me a question about something you know I have the answer too, and I'll waste the rest of the day talking to you about it, if you'll let me.

I'm not always great at seeing the big picture, and until recently it had never occurred to me that I might enjoy mentoring other programmers.

Room 11 on Stackoverflow is where I ... I think the right word is roost, or possibly fester ...

There are an amazing bunch of internals and community programmers that frequent the room. It has produced, indirectly, many great projects, and some of the most popular RFC's.

I think this is a pretty widely known fact by now, and as a result, the room manages to attract new internals and extension developers.

One such programmer has just released, to roaring crowds (I think), a very nice data structures extension for PHP 7.

Over the last few weeks, Rudi and I have collaborated in Room 11, and by other means, but, for the first time, I didn't take the lead writing code.

I answered questions in as much detail as my skills allow, helped Rudi debug, reviewed code, made a few very minor fixes, and finally helped him prepare to publish his work.

I enjoyed this experience at least as much as I would have enjoyed writing the code myself, maybe more.

Rudi has kindly, without me asking him too, put my name in the credits, he's also thanked me no end.

All I actually did, was stuff I would have done anyway, stuff I enjoy, stuff that actually is less demanding and better for me than always trying to pursue solutions for myself.

This is absolutely a revelation to me, and I think we all see within it a way to keep the dragon at bay.

I look forward to the next project, and the next programmer ... is it you ?

Consider this an invitation.

Saturday, 9 January 2016

Internal Law

Fig 1. Lady Justice
For all of the last two decades, internals has functioned without any kind of agreement between participants regarding how we should conduct discourse, or how we should represent the project in the community.

Recently, our good friend Anthony Ferrara submitted a request for comments on a draft proposal that we should adopt a code of conduct.

In short, it proposes that we define a contributor agreement, to which every contributor must agree. In addition, it proposes that we nominate by election a committee charged with investigating breaches of the agreement, and if necessary taking action.

Note that nobody is voting on this next week; This is a request for comments, on a draft proposal.

Anecdotes aren't evidence, but they can be amusing, and it's the weekend ...

I have two children of primary school age, 6 and 11. Parents evening is when we go into the school and check out their report cards, this years work, and see how they are progressing.

An evening filled with pride, and hilarity ...

Fig 2. The boy's first self review, aged 4 (writing by teacher).

At the end of the evening, the children are asked to sign a code of conduct. In it they promise to be nice to other students, treat everyone the same, abide by the school rules, behave when they are in school uniform as if they are in school.

They have a cumulative age that is less than the age of our youngest contributor (as far as I'm aware). They sign it every year.

They communicate to us and their teachers, that they understand why it's a good idea to enter into such an agreement. They both enjoy school, hugely, for all the right reasons. They understand that to maintain an enjoyable atmosphere, certain things are required of them. They even understand the implications of their wearing a uniform on the way to and from school, and on school trips.

I am therefore baffled by a minority of very vocal members of the community rebelling against the core idea, and every bit of content in the draft proposal.

I don't care for the wording of the proposed code of conduct used in the draft. Specifically, I object to the idea that we should try to limit "offence" ...


Certain things happen to be offensive to most sensibilities, things like racially motivated derogatory language, or sexist language. However these things aren't wrong because they are offensive, they are wrong because of their motivation, they only happen to be offensive, to most sensibilities.

Some people have said that this isn't needed because they can't think of a time it could have been used to improve any situation. This seems to be argumentum ad ignorantiam, as other members of the community disagreed. Even if it were true that it would never have been useful for anyone, that is not a good reason to argue that we don't need it.

Much of the world has laws not too dissimilar from those rules proposed, laws governing public and online decency and conduct. We all must agree to them, we have no choice, we can't back out because we think our freedom of expression is being encroached upon, or because we don't trust governance or policing. We can't not participate in society and so avoid being subject to such law.

The aim to provide the same kind of protection for participants in PHP regardless of their country of origin, or anything else, seems like a laudable one.

Maybe, that would have been a better starting place than the one we had ... 

Internals has become a more political place, as a result of the idea to vote on things like new features, and release managers. Serving the interests of the majority has done pretty cool things in recent years, so nobody can say in good conscience that the apparent politicization is a bad thing. 

The way to effect politics is not to stand at the sidelines and shout, it is to get involved. So while I have my own doubts about how effective the committee charged with investigating breaches can be, I'll volunteer my own time for that, before I'll say we shouldn't have it because of my own doubts.

So, the wording needs tweaking and the process needs rational discussion ... All the while we should remember that there are 5 year old children that understand why this is a good idea in principle.

Friday, 20 November 2015

APC and Me

Fig1. An APC logo.

When it was decided that Zend's Optimizer Plus would be merged into PHP, APC was already in a pretty poor state, there hadn't been a stable release for quite some time.

We were moving towards having a built in (abandoned in php-src/ext) opcode caching solution but it was not obvious that APC was going to keep being maintained.

I think all of us assumed that the bugs that were being experienced were entirely down to the opcode caching parts of APC, since they were the most complex and most frequently altered parts of APC.

So, one day (few nights), I stripped APC of opcode caching.

One of the things I decided I would tidy up was the implementation and usage of locking.

The supported kinds of locks are:
  • File locking
  • Mutex
  • Read/Write Locks
  • Spin Locks
Brief explanations follow.

File Locking


I'm going to assume that everybody reading knows what this is, and even without experience, can sense that it is probably the most inferior in the list.

I sincerely hope that nobody uses file locking today.

The reason it exists is because people deploy PHP in all kinds of places, places we don't get to hear about until something goes wrong. Those places might not have support for anything other than file locking, so it stays.

Mutex


This is your most basic kind of synchronization. Mutex means Mutual Exclusion, so we know that this kind of synchronization is exclusive.

This was the default locking for APC

Read/Write Locks


Read/Write locking allows a shared lock to be acquired for reading, this means many readers can be supported without exclusion. An exclusive lock is only required for writing.

This is the default for APCu

Spin Locks


Speaking as a programmer who spends a lot of time writing multi-threaded code, in various languages: A spin lock is about the worst kind of synchronization imaginable, it is basically a predicated busy wait loop.

This remains in APCu for the same reason file locking does, and I actually have heard of people using it and can't convince them to do otherwise.


Scary Things


I didn't know the APC code base before that first night, had never read any of it before.

Whenever a programmer reads code from a prospective project that another programmer wrote, they have criticisms. Some of them are just our ego talking, some of them are wrong, some of them are probably a mechanism to motivate us to keep working. Hardly any of them are worth mentioning, or doing anything about.

The task at hand is to get the thing working, not fix every problem that nobody ever had.

So I cleaned up, I worked on it for a few days and pushed it to pecl.


OMG


So, this blog post started out as something completely different.

APCu was demonstrably unstable, and I thought this was because it broke a basic rule. It seemed to acquire a read lock, when it should be acquiring a write lock.

There seemed to be a clear path to race conditions, in other words.

While trying to find the code in the APC code base that caused the problem I made this search: apc_cache_release

The APCu version omits atomicity, it omits safety; In my haste to clean up, I have made a horrible mistake.

There was even a pull request, from one of the elders of PHP, that I chose not to merge, for three years.

The worst thing about being wrong is that I inevitably feel dumb as rocks.

The best thing about being wrong is things that didn't make sense before, like this, or this, and many other bugs besides, start to make sense.

So, I've found the reason that APCu was unstable, it was me.

This probably caused a problem for a fairly large number of people.

Sorry about that, I'll get to fixing it ...

I'm afraid I don't have time to blog now, too much work to do :(