[GSoC 2014] Http Server Proposal

newer
GSoC 2014 proposal for...

Vinícius dos Santos Oliveira

1 Mar 2014 1 Mar '14

1:06 a.m.

Hi guys, I'm interested in develop an asynchronous HTTP server for Boost as part of GSoC. The design would be inspired from Node.js without the facilities or inspirations that were created to overcome JavaScript quirks and limitations. The library would be modular to the point where you can use only small parts of it and you could also replace components that you don't like. The library should have a small overhead to be used on embedded projects, but the modular design will allow you to use higher abstractions when such concerns don't exist. Of course it's difficult to find the "right" answer to the problem, then I'll limit the higher-abstractions to "proven" techniques and avoid controversial features. I hope to give more details in the middle of the next week. The library will not be about writing template-driven web pages, but it will be about HTTP and you will be able to do things like video live stream via HTTP (try to do that with Django or PHP). I also intend to allow a efficient use in multithreaded environment with high-load of data (although network applications are usually I/O bound and not CPU bound). Of course it'll be possible to use WebSocket with this library too. Another thing I want to allow is to mock HTTP requests and transparently replace parts of the stack without affecting already written code. The purpose of this feature is to allow other transport mechanism such as FastCGI and also provide a future-proof approach by better allowing HTTP/2.0 when released. I would implement such library on top of Boost ASIO and I'd include facilities that I'm sure would work on long-term. Because the success of such project is very sensible to the chosen design, I'd limit its scope to things that I know it will work long-term. If the implementation project end up being too small, I can also implement WebSocket support (I had implemented it before and it wouldn't be such challenge for me), where the implementation effort is higher than the interface effort. I'm not sure about the deliverables (yet!), but I hope to give more details soon, once I have written nice texts. I'm particularly interested in an elegant way to represent some data structures, but I'll detail the plans soon (lots of text to write, ideas to explain and references to convince, among other). The text was a bit long, but it's because I already have something in mind. I hope to deliver a proposal explaining all the design, why I think it would work and other matters in the middle of the next week. I'm also open to suggestions and modifications in the not-yet-presented-proposal. I can provide background about my experiences later, but for now I want to know what do you guys think about the idea. TL;DR: For now I want to know what do you guys think about having a core HTTP server library in Boost. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Attachments:

signature.asc (application/pgp-signature — 836 bytes)

Show replies by date

Niall Douglas

1 Mar 1 Mar

11:46 a.m.

On 28 Feb 2014 at 22:06, Vinícius dos Santos Oliveira wrote:

...

I'm interested in develop an asynchronous HTTP server for Boost as part of GSoC.

I would implement such library on top of Boost ASIO and I'd include

For something as "chunky" as as a HTTP server, you might consider using proposed Boost.AFIO which itself is built atop Boost.ASIO. AFIO will take care of the asynchronous file access for you plus it's far quicker at file access on Windows than the Win32 API is, plus the batch facilities would make implementing HTTP pipelining far easier. You would still have a lot of work remaining e.g. session persistence, but it wouldn't be as hard. Algorithmically an async HTTP server is not challenging. Look into Python's Twisted. Do what they do. Disclaimer: I designed proposed Boost.AFIO. Niall -- Currently unemployed and looking for work in Ireland. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Vinícius dos Santos Oliveira

8:37 p.m.

Em Sáb, 2014-03-01 às 11:46 +0000, Niall Douglas escreveu:

...

Algorithmically an async HTTP server is not challenging.

Yes, I know that. The most challenging part in a HTTP server (in my opinion) is the interface, not event the implementation. That's why I'm considering WebSocket support too, but I have to investigate further about the full scope of the proposal.

...

Look into Python's Twisted. Do what they do.

I studied several web solutions to serve as inspiration, but I haven't checked Python's Twisted or Boost.AFIO. I'll surely check them now. Thanks for pointing me them out. It may improve my proposal already. :)

...

Disclaimer: I designed proposed Boost.AFIO.

Then I know who I shall bug for questions. :) -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Bjorn Reese

6 Mar 6 Mar

11:23 a.m.

On 03/01/2014 02:06 AM, Vinícius dos Santos Oliveira wrote:

...

I'm interested in develop an asynchronous HTTP server for Boost as part of GSoC.

I think that this would be a good project, and you look like a capable candidate judging from your descriptions and your Tufao project. A reference to other readers: https://github.com/vinipsmaker/tufao The two leading contenders for C++ HTTP servers based on Boost.Asio are Pion and cpp-netlib, and you should investigate them: https://github.com/cloudmeter/pion http://cpp-netlib.org/ Back in 2008 there was talk about merging these libraries into a Boost.Network proposal, but I do not know what happened to these efforts.

...

The library would be modular to the point where you can use only small parts of it and you could also replace components that you don't like.

This is a very sound design principle. Two obvious candidates for independent components are HTTP and URL parsing/generation. As the HTTP component would have to handle HTTP/1.1 chunking and WebSockets, it may be an idea to design it as a streaming parser as it provides a unified interface for all HTTP artifacts, including the live video streaming that you mention. Another use case to consider is the UPnP HTTPU (HTTP over UDP.) Concerning the URL parser/generator, an URI library has been proposed for the C++ standard: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3625.html

...

to things that I know it will work long-term. If the implementation project end up being too small, I can also implement WebSocket support

I would worry more about it becoming too large ;-)

Vinícius dos Santos Oliveira

4:34 p.m.

I'm almost finishing the initial proposal and I hope to send it to you guys today. Em Qui, 2014-03-06 às 12:23 +0100, Bjorn Reese escreveu:

...

The two leading contenders for C++ HTTP servers based on Boost.Asio are Pion and cpp-netlib, and you should investigate them:

https://github.com/cloudmeter/pion http://cpp-netlib.org/

I've seen mentions to the cpp-netlib before, but I'll take a closer look now. About the other one (pion), I don't remember/know. I'll check it too.

...

As the HTTP component would have to handle HTTP/1.1 chunking and WebSockets, it may be an idea to design it as a streaming parser as it provides a unified interface for all HTTP artifacts, including the live video streaming that you mention.

Agreed.

...

Another use case to consider is the UPnP HTTPU (HTTP over UDP.)

The proposal is already too demanding and support for UPnP HTTPU is just a matter of implementing a new backend. Maybe I'm not seeing some special need here and I'm wrong. In this case, just correct me. Even so, it's difficult for you guys to tell whether the proposal is enough or not when I didn't even published it. I hope to change that in the following hours of this current day.

...

Concerning the URL parser/generator, an URI library has been proposed for the C++ standard:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3625.html

I'm following the work of the responsible working group, but thanks for bringing this to our attention. ;-)

...

I would worry more about it becoming too large ;-)

I can always implement the controversial parts and only propose a subset to be merged within boost, while the whole proposal reaches more maturity. =) -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Cory Nelson

5:28 p.m.

On Fri, Feb 28, 2014 at 7:06 PM, Vinícius dos Santos Oliveira wrote:

...

Hi guys,

I'm interested in develop an asynchronous HTTP server for Boost as part of GSoC.

The design would be inspired from Node.js without the facilities or inspirations that were created to overcome JavaScript quirks and limitations.

The library would be modular to the point where you can use only small parts of it and you could also replace components that you don't like.

The library should have a small overhead to be used on embedded projects, but the modular design will allow you to use higher abstractions when such concerns don't exist. Of course it's difficult to find the "right" answer to the problem, then I'll limit the higher-abstractions to "proven" techniques and avoid controversial features. I hope to give more details in the middle of the next week.

The library will not be about writing template-driven web pages, but it will be about HTTP and you will be able to do things like video live stream via HTTP (try to do that with Django or PHP). I also intend to allow a efficient use in multithreaded environment with high-load of data (although network applications are usually I/O bound and not CPU bound). Of course it'll be possible to use WebSocket with this library too.

Another thing I want to allow is to mock HTTP requests and transparently replace parts of the stack without affecting already written code. The purpose of this feature is to allow other transport mechanism such as FastCGI and also provide a future-proof approach by better allowing HTTP/2.0 when released.

I would implement such library on top of Boost ASIO and I'd include facilities that I'm sure would work on long-term. Because the success of such project is very sensible to the chosen design, I'd limit its scope to things that I know it will work long-term. If the implementation project end up being too small, ...

Making a good, correct HTTP 1.1 parser is very non-trivial -- this is actually a pretty ambitious project for GSOC. I'd recommend condensing your idea as much as possible and working on it in cleanly separable steps. Don't get distracted by Websockets and FastCGI! Please make the parser I/O agnostic and non-blocking (push bytes in, get a chunk of a http message as a result or request for more data) -- it'll be simpler to test and will allow porting to lower-level I/O when asio becomes too slow. -- Cory Nelson http://int64.org

Dean Michael Berris

11 p.m.

Hi Everyone, I'm Dean and I'm the co-maintainer of cpp-netlib. I would like to say thank you first for bringing cpp-netlib into this discussion. I'm excited that some people still both remember the library and actually that there was an effort to get it and Pion to become Boost.Network. The current stable release of cpp-netlib (0.11.0) is actually the last release where the library lives under the boost::network namespace. There's a full HTTP/1.0 server implementation there, and there's significant work to be done to support all of HTTP/1.1. That said if someone were willing to mentor turning cpp-netlib 0.11.0 into Boost.Network officially, I would very much be willing to help in that process (at a consultative/adviser/mentor(maybe) role). Because the code is laid out to be Boost-compliant in 0.11.0, it may be possible to bring back the Boost.Build shims and have a fork on github maintained by someone else that actually brings it into Boost library standards. Unfortunately the project is moving into a direction where we're doing mostly a rewrite for C++11/C++14 compliance, and as a proving ground for some of the standardization efforts Glyn Matthews (co-maintainer) and I would like to push forward with. The next release which we're hoping to do at the end of 2014 will be moving out of the boost::network namespace and has a very different code structure from Boost's. We've also been trying to minimize the reliance on some Boost libraries that we use for implementation, for a few reasons -- none of which is a reflection on Boost's state, but rather more for project organizational reasons. For example we've recently migrated all our tests away from Boost.Test and are working with igloo and Google's Test/Mock libraries. We're also moving to using Asio directly (not the Boost.Asio version). However as mentioned if someone would like to take cpp-netlib's 0.11.0 version and turn it into a proper Boost.Network library, I'm more than happy to make myself available for Hangouts to get the process going as quickly as possible. Thank you for your interest, and I wish you have great success in GSoC! On Fri, Mar 7, 2014 at 4:28 AM, Cory Nelson wrote:

...

On Fri, Feb 28, 2014 at 7:06 PM, Vinícius dos Santos Oliveira wrote:

...
Hi guys,

I'm interested in develop an asynchronous HTTP server for Boost as part of GSoC.

The design would be inspired from Node.js without the facilities or inspirations that were created to overcome JavaScript quirks and limitations.

The library would be modular to the point where you can use only small parts of it and you could also replace components that you don't like.

The library should have a small overhead to be used on embedded projects, but the modular design will allow you to use higher abstractions when such concerns don't exist. Of course it's difficult to find the "right" answer to the problem, then I'll limit the higher-abstractions to "proven" techniques and avoid controversial features. I hope to give more details in the middle of the next week.

The library will not be about writing template-driven web pages, but it will be about HTTP and you will be able to do things like video live stream via HTTP (try to do that with Django or PHP). I also intend to allow a efficient use in multithreaded environment with high-load of data (although network applications are usually I/O bound and not CPU bound). Of course it'll be possible to use WebSocket with this library too.

Another thing I want to allow is to mock HTTP requests and transparently replace parts of the stack without affecting already written code. The purpose of this feature is to allow other transport mechanism such as FastCGI and also provide a future-proof approach by better allowing HTTP/2.0 when released.

I would implement such library on top of Boost ASIO and I'd include facilities that I'm sure would work on long-term. Because the success of such project is very sensible to the chosen design, I'd limit its scope to things that I know it will work long-term. If the implementation project end up being too small, ...

Making a good, correct HTTP 1.1 parser is very non-trivial -- this is actually a pretty ambitious project for GSOC. I'd recommend condensing your idea as much as possible and working on it in cleanly separable steps. Don't get distracted by Websockets and FastCGI!

Please make the parser I/O agnostic and non-blocking (push bytes in, get a chunk of a http message as a result or request for more data) -- it'll be simpler to test and will allow porting to lower-level I/O when asio becomes too slow.

-- Cory Nelson http://int64.org

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Vinícius dos Santos Oliveira

7 Mar 7 Mar

7:25 a.m.

I'm sorry for taking so long to present my proposal, but some unpredictable events delayed the available time I had to work on the proposal and the large number of mentions you guys made turned my work into a task more difficult. Anyway, like some people advocating on open source projects state, "release early, release often". Then I'll post the link to the incomplete proposal through github, then I'll have a nicely formatted MarkDown file and you guys can easily use the diff button on the web interface to see the changes: https://github.com/vinipsmaker/gsoc2014-boost Em Qui, 2014-03-06 às 11:28 -0600, Cory Nelson escreveu:

...

Making a good, correct HTTP 1.1 parser is very non-trivial -- this is actually a pretty ambitious project for GSOC. I'd recommend condensing your idea as much as possible and working on it in cleanly separable steps. Don't get distracted by Websockets and FastCGI!

I haven't said anything about timelines yet. I want to have a clear core library and then move on to measure/guess what can be implemented during the GSoC and define the timeline.

...

Please make the parser I/O agnostic and non-blocking (push bytes in, get a chunk of a http message as a result or request for more data) -- it'll be simpler to test and will allow porting to lower-level I/O when asio becomes too slow.

About the parser, it's less useful for people than an actual http server. I was planning to initially use Ryan Dahl's HTTP parser, then I can focus on imediate needs. A parser with a good interface could be provided in the future. But... I'm too late and Vinnie Falco just suggested the same: Em Qui, 2014-03-06 às 09:44 -0800, Vinnie Falco escreveu:

...

These are good guidelines. Let me point out that the HTTP parser from node.js is permissively licensed, and incredibly robust:

https://github.com/joyent/http-parser

It's hardened and battle-tested so there are no exploits in it. I use it in my own HTTP message wrapper and handshaking logic. Instead of writing one yourself, I highly recommend that you instead take this http-parser as is and put a boost-worthy C++ front end around it.

The problem with the Ryan Dahl's HTTP parser is that it doesn't support arbitrary HTTP methods/verbs. Then it should be replaced in the future. Em Sex, 2014-03-07 às 10:00 +1100, Dean Michael Berris escreveu:

...

[...] However as mentioned if someone would like to take cpp-netlib's 0.11.0 version and turn it into a proper Boost.Network library, I'm more than happy to make myself available for Hangouts to get the process going as quickly as possible.

The scope of the cpp-netlib is quite large. I'd like to focus on the HTTP server abstraction. But thanks for the support anyway, Dean. I might bug you for questions regarding the cpp-netlib's design. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

isundill

10:30 a.m.

I'm very interested in this project, but I'm not eligible for the Gsoc. I would be glad to help you working on this project. --- Regards, Thibault Barre Le 07-Mar-14 10:25 AM, Vinícius dos Santos Oliveira a écrit :

...

I'm sorry for taking so long to present my proposal, but some unpredictable events delayed the available time I had to work on the proposal and the large number of mentions you guys made turned my work into a task more difficult.

Anyway, like some people advocating on open source projects state, "release early, release often". Then I'll post the link to the incomplete proposal through github, then I'll have a nicely formatted MarkDown file and you guys can easily use the diff button on the web interface to see the changes: https://github.com/vinipsmaker/gsoc2014-boost

Em Qui, 2014-03-06 às 11:28 -0600, Cory Nelson escreveu:

...
Making a good, correct HTTP 1.1 parser is very non-trivial -- this is actually a pretty ambitious project for GSOC. I'd recommend condensing your idea as much as possible and working on it in cleanly separable steps. Don't get distracted by Websockets and FastCGI!

I haven't said anything about timelines yet. I want to have a clear core library and then move on to measure/guess what can be implemented during the GSoC and define the timeline.

...
Please make the parser I/O agnostic and non-blocking (push bytes in, get a chunk of a http message as a result or request for more data) -- it'll be simpler to test and will allow porting to lower-level I/O when asio becomes too slow.

About the parser, it's less useful for people than an actual http server. I was planning to initially use Ryan Dahl's HTTP parser, then I can focus on imediate needs. A parser with a good interface could be provided in the future. But... I'm too late and Vinnie Falco just suggested the same:

Em Qui, 2014-03-06 às 09:44 -0800, Vinnie Falco escreveu:

...
These are good guidelines. Let me point out that the HTTP parser from node.js is permissively licensed, and incredibly robust:

https://github.com/joyent/http-parser

It's hardened and battle-tested so there are no exploits in it. I use it in my own HTTP message wrapper and handshaking logic. Instead of writing one yourself, I highly recommend that you instead take this http-parser as is and put a boost-worthy C++ front end around it.

The problem with the Ryan Dahl's HTTP parser is that it doesn't support arbitrary HTTP methods/verbs. Then it should be replaced in the future.

Em Sex, 2014-03-07 às 10:00 +1100, Dean Michael Berris escreveu:

...
[...] However as mentioned if someone would like to take cpp-netlib's 0.11.0 version and turn it into a proper Boost.Network library, I'm more than happy to make myself available for Hangouts to get the process going as quickly as possible.

The scope of the cpp-netlib is quite large. I'd like to focus on the HTTP server abstraction.

But thanks for the support anyway, Dean. I might bug you for questions regarding the cpp-netlib's design.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Vinícius dos Santos Oliveira

3:58 p.m.

Em Sex, 2014-03-07 às 13:30 +0300, isundill escreveu:

...

I'm very interested in this project, but I'm not eligible for the Gsoc. I would be glad to help you working on this project.

Help and feedback is always appreciated. Thanks for the support, Thibault. =) -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Jens Weller

5:28 p.m.

A little help might be this image of how a HTTP Server should behave: https://raw.github.com/for-GET/http-decision-diagram/master/httpdd.png kind regards, Jens Weller

Niall Douglas

5:46 p.m.

On 7 Mar 2014 at 4:25, Vinícius dos Santos Oliveira wrote:

...

I'm sorry for taking so long to present my proposal, but some unpredictable events delayed the available time I had to work on the proposal and the large number of mentions you guys made turned my work into a task more difficult.

Firstly, this is a not bad initial design given your unfamiliarity with Boost. Its biggest problem is vagueness in some key areas, but as a review of the existing approaches it's not bad. I'll just make some observations: * ASIO can use futures or callbacks. It can use threads or coroutines. Error handling can be an error code or exceptions. In short: ASIO is very flexible. The N3964 paper simply seeks to standardise ASIO's already existing design, so all those facilities are already there and available. * AFIO is probably not hugely useful except for doing async file i/o which ASIO can't do, despite what ASIO claims in its docs. In other words you'd bring in AFIO for the static content backend where for example you can enumerate directories asynchronously, otherwise AFIO is overkill. * I would very seriously reduce the scope of your present proposal. Just to write a semi-decent fully asynchronous static content HTTP 1.1 server is more than enough to consider right now. I'd forget about any CGI or anything else like that for now. I'd *even* suggest restricting scope to HTTP 1.0 support as your first milestone, and then slowly extend out into HTTP 1.1. * You mentioned "The implementation of a HTTP server isn't very challenging". Algorithmically you're right, but the fact people keep reinventing HTTP servers shows how hard implementing a *good* HTTP server is. HTTP servers are a good example of why a compsci whiz or an algorithms whiz can make lousy "real world" programmers. The hard part of a decent HTTP server is getting to a 100% asynchronous internal engine which is exception safe and linear scalable to load. I would *not* underestimate how hard one of these is: I would personally estimate about 1000 hours for me to implement just a fully async HTTP 1.0 static content server with validation and test suite and Boost quality documentation. If you think that proposed Boost.AFIO - which is just 1000 active LOC by the way - has consumed about 600 hours, and it's a fraction of the complexity of a HTTP 1.0 server, you'll see what I mean. Good luck! Niall -- Currently unemployed and looking for work in Ireland. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Vinícius dos Santos Oliveira

6:47 p.m.

Em Sex, 2014-03-07 às 17:46 +0000, Niall Douglas escreveu:

...

* I would very seriously reduce the scope of your present proposal. Just to write a semi-decent fully asynchronous static content HTTP 1.1 server is more than enough to consider right now. I'd forget about any CGI or anything else like that for now. I'd *even* suggest restricting scope to HTTP 1.0 support as your first milestone, and then slowly extend out into HTTP 1.1.

* You mentioned "The implementation of a HTTP server isn't very challenging". Algorithmically you're right, but the fact people keep reinventing HTTP servers shows how hard implementing a *good* HTTP server is. HTTP servers are a good example of why a compsci whiz or an algorithms whiz can make lousy "real world" programmers. The hard part of a decent HTTP server is getting to a 100% asynchronous internal engine which is exception safe and linear scalable to load. I would *not* underestimate how hard one of these is: I would personally estimate about 1000 hours for me to implement just a fully async HTTP 1.0 static content server with validation and test suite and Boost quality documentation. If you think that proposed Boost.AFIO - which is just 1000 active LOC by the way - has consumed about 600 hours, and it's a fraction of the complexity of a HTTP 1.0 server, you'll see what I mean.

The main reason for my attention for CGI, multithread and so many other features is to get a core design that works. I'm not planning to deliver all features (I didn't write the timeline yet). Another reason to cite so many features is to raise discussions (like this) and to help with the arguments. And my plan to get a "good" HTTP server is very simple: (1) expose the HTTP power and (2) allow different backends (3) under a performant and flexible API. But you're right about a small set of core features that is (1) 100% asynchronous, (2) exception safe, (3) linear scalable with (4) validation, (5) test suite and (6) Boost quality documentation. My plan to accelerate the development, like stated in the current version of the proposal, is to reuse a HTTP parser being used heavily under production around the world. And I wouldn't be interested in Boost if I was expecting an easy task. I'll spend the rest of the day finishing the "vagueness in some key areas" that you mentioned. Thanks for the detailed feedback. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Vinícius dos Santos Oliveira

8 Mar 8 Mar

1:54 a.m.

Em Sex, 2014-03-07 às 04:25 -0300, Vinícius dos Santos Oliveira escreveu:

...

Anyway, like some people advocating on open source projects state, "release early, release often". Then I'll post the link to the incomplete proposal through github, then I'll have a nicely formatted MarkDown file and you guys can easily use the diff button on the web interface to see the changes: https://github.com/vinipsmaker/gsoc2014-boost

I didn't finish the proposal yet, but I made several assorted improvements today. Now the proposal isn't terribly vague compared to how it was before, but there are still lots of improvements to do. Github'll even help you with a nice interface comparing the changes (and still offer the possibility to nicely render the MarkDown files): https://github.com/vinipsmaker/gsoc2014-boost/compare/37811f06496645c8f8d62c... I don't want to spam the inbox of you guys, then I'll only send this email regarding updates to the proposal and one more once I think it's complete. For you guys wishing to receive more notifications, there is the possibility to subscribe to the project's changes clicking on the "watch" button through the github page. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Niall Douglas

2:30 p.m.

On 7 Mar 2014 at 22:54, Vinícius dos Santos Oliveira wrote:

...

I don't want to spam the inbox of you guys, then I'll only send this email regarding updates to the proposal and one more once I think it's complete. For you guys wishing to receive more notifications, there is the possibility to subscribe to the project's changes clicking on the "watch" button through the github page.

Can I clarify Vinicius: this is a GSoC proposal right? Niall --- Boost C++ Libraries Google Summer of Code 2014 admin https://svn.boost.org/trac/boost/wiki/SoC2014

Vinícius dos Santos Oliveira

2:58 p.m.

Em Sáb, 2014-03-08 às 14:30 +0000, Niall Douglas escreveu:

...

Can I clarify Vinicius: this is a GSoC proposal right?

Officially it's not a proposal yet. It'll be next week when the application period start. I think. I'm just excited about the idea and I'm spending some effort to make a great proposal. I think feedback is important to achieve the awesomeness level of a proposal. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Niall Douglas

4:11 p.m.

On 8 Mar 2014 at 11:58, Vinícius dos Santos Oliveira wrote:

...

...
Can I clarify Vinicius: this is a GSoC proposal right?

Officially it's not a proposal yet. It'll be next week when the application period start. I think.

Did you have a specific mentor in mind for this proposal? I ask, because a strong proposal - which you already have now - also needs a proposed mentor to gain GSoC funding. Moreover, that proposed mentor can help you whittle your current proposal into a suitable summer project work item, and I'd like to match you up as soon as possible. Niall --- Boost C++ Libraries Google Summer of Code 2014 admin https://svn.boost.org/trac/boost/wiki/SoC2014

Vinícius dos Santos Oliveira

5:18 p.m.

Em Sáb, 2014-03-08 às 16:11 +0000, Niall Douglas escreveu:

...

Did you have a specific mentor in mind for this proposal?

Not yet.

...

I ask, because a strong proposal - which you already have now - also needs a proposed mentor to gain GSoC funding.

GSoC gives a certain degree of freedom for each organization and the "rules" aren't general to all projects. I'm not used to boost GSoC "rules". I thought that this was a valid proposal, because the "Ideas/GSoC" page on the boost wiki mentioned the "Protocol implementations for Boost.Asio" item. I didn't noticed such warnings on the other pages (submission template, hints) also. Now that you mentioned such need to have a proposed mentor earlier than I expected, I became a little more worried.

...

Moreover, that proposed mentor can help you whittle your current proposal into a suitable summer project work item, and I'd like to match you up as soon as possible.

If it's not a problem for you, I'd like to have your help in whittle my proposal. I just want to finish the descriptions/designs of a usable core set of components before the decision of a timeline and deliverables. Do you think it's a good idea? -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Niall Douglas

6:50 p.m.

On 8 Mar 2014 at 14:18, Vinícius dos Santos Oliveira wrote:

...

...
I ask, because a strong proposal - which you already have now - also needs a proposed mentor to gain GSoC funding.

Now that you mentioned such need to have a proposed mentor earlier than I expected, I became a little more worried.

I should quickly explain how GSoC selects projects. Firstly students submit their proposals. We, the Boost community, then review and rank the proposals. The rank ordered proposals go to Google who then usually choose the top X proposals where X is the number of slots they award. The community ranking stage you see goes much better for you if you have a mentor championing your proposal to the community during the review stage. Without a mentor, you have no such champion.

...

...
Moreover, that proposed mentor can help you whittle your current proposal into a suitable summer project work item, and I'd like to match you up as soon as possible.

If it's not a problem for you, I'd like to have your help in whittle my proposal.

Well, I'll be blunt in saying my main interest in your proposal is extra testing for AFIO. I'm especially keen on ironing out ASIO-AFIO interop kinks in real world code. I'm also not as expert in HTTP and web services as others on this list who would therefore be better mentors. Besides, I also have a fair workload in administrating GSoC, it's turning out to be quite a bit at the moment.

...

I just want to finish the descriptions/designs of a usable core set of components before the decision of a timeline and deliverables. Do you think it's a good idea?

Oh yes, keep going. Potential mentors are watching everything you do with interest. Just remember to be realistic when scheduling: you don't get much done in a summer project. Tip: look more deeply into Pion which is a HTTP implementation based on Boost.ASIO, it's an alternative to cpp-netlib and it's also Boost based. Niall --- Boost C++ Libraries Google Summer of Code 2014 admin https://svn.boost.org/trac/boost/wiki/SoC2014

Bjorn Reese

9 Mar 9 Mar

2:52 p.m.

On 03/08/2014 02:54 AM, Vinícius dos Santos Oliveira wrote:

...

...
https://github.com/vinipsmaker/gsoc2014-boost

This is a strong and well-researched proposal. I volunteer as a mentor for this project. I have some suggestions for improving your proposal. Do not let the sheer volume discourage you; you are definitely on the right track. List the overall goals more clearly at the start. For example: 1. Targets embeddable HTTP servers (e.g. to implement ReST APIs). These servers should be able to co-exist with other embeddable communication facilities (e.g. Bonjour device discovery or BitTorrent.) 2. Create a C++ toolbox for embeddable HTTP servers. This requires a modular design and a coherent thread model. 3. Flexible design that can encompass HTTP chunking, HTTP pipelining, WebSockets, FastCGI, etc. 4. Scalable performance. This requires an asynchronous design. 5. Reuse existing std or Boost components, such as Boost.Asio. You should introduce the overall design (messages, connections, and backends) before discussing performance. Otherwise, your performance discussion is going to be hard to follow. You should also describe how the distinction between header and body, and especially how streamable bodies (HTTP chunking and WebSockets), fit into the design. Most of the performance discussion should be moved to a separate section. For instance, I do not need to know about an extra level of indirection this early in the proposal. Good discussion about the other HTTP servers. I have the following comments though: The discussion on Twisted is inappropriate for a proposal. You should either investigate Twisted or remove the section. Furthermore, you should look into Pion. The proposal does not mention Request-URIs, and thus makes no reference to the C++ standard proposal N3625. Concerning your HTTP server design, you currently have a server backend that the user must implement to obtain HTTP requests via the async_handle() functions. This makes the user a passive party in the design, who only has to react to incoming requests. I suggest that you consider a design that is closer to the Boost.Asio design. Let the user become the active party, who asks explicitly for the next request. This can be done by adding async_send() and async_receive() functions on the http::server::server class. These functions should handle partial transmission under the hood, so their handlers are only trigger when an entire HTTP request has been received (or a response has been sent.) This will render the backend superfluous.

Niall Douglas

7:58 p.m.

On 9 Mar 2014 at 15:52, Bjorn Reese wrote:

...

...
...
https://github.com/vinipsmaker/gsoc2014-boost

This is a strong and well-researched proposal. I volunteer as a mentor for this project.

Outstanding, thank you Bjorn. Vinícius, Bjorn is one of the best mentors you could possibly have for this project (feel free to research what open source he has contributed to over the past fifteen years, you'll see what I mean), well done on getting him interested. Niall --- Boost C++ Libraries Google Summer of Code 2014 admin https://svn.boost.org/trac/boost/wiki/SoC2014

Vinícius dos Santos Oliveira

10 Mar 10 Mar

3:18 a.m.

Em Dom, 2014-03-09 às 15:52 +0100, Bjorn Reese escreveu:

...

This is a strong and well-researched proposal. I volunteer as a mentor for this project.

Great. And thanks for the detailed feedback.

...

List the overall goals more clearly at the start.

Good idea. Now the proposal has some points before the design consideration at the very beginning of the text. Maybe I can add a small paragraph detailing why this proposal is important to boost too.

...

For example: [...]

I added this list to my proposal at https://github.com/vinipsmaker/gsoc2014-boost#some-target-scenarios . I hope you don't mind (text is quoted anyway). It seems you understood my proposal very well.

...

You should introduce the overall design (messages, connections, and backends) before discussing performance. Otherwise, your performance discussion is going to be hard to follow. You should also describe how the distinction between header and body, and especially how streamable bodies (HTTP chunking and WebSockets), fit into the design.

I'm working on that.

...

Most of the performance discussion should be moved to a separate section. For instance, I do not need to know about an extra level of indirection this early in the proposal.

Done.

...

Good discussion about the other HTTP servers. I have the following comments though: The discussion on Twisted is inappropriate for a proposal. You should either investigate Twisted or remove the section.

Done

...

Furthermore, you should look into Pion.

I'm working on that.

...

The proposal does not mention Request-URIs, and thus makes no reference to the C++ standard proposal N3625.

I'll fix it soon.

...

Concerning your HTTP server design, you currently have a server backend that the user must implement to obtain HTTP requests via the async_handle() functions.

Actually, the documented design is pretty vague yet. It doesn't make much sense to double document here on the list and there on the MarkDown file, then I'll be brief now and work there. I can warn about the update later. So... briefly, the backend is not meant to register user callbacks. It's meant to abstract different communication mechanisms (builtin HTTP server, FastCGI, CoAP, ...) to the request and response objects. It could be used to obtain new HTTP requests too, but I'll have to think more about this when it's time to detail http+https+threaded design (the user might be interested in use the same handlers for all backends).

...

This makes the user a passive party in the design, who only has to react to incoming requests. I suggest that you consider a design that is closer to the Boost.Asio design. Let the user become the active party, who asks explicitly for the next request. This can be done by adding async_send() and async_receive() functions on the http::server::server class.

Looks like you're worried about an active design that is closer to ASIO. I'll keep that in mind.

...

These functions should handle partial transmission under the hood, so their handlers are only trigger when an entire HTTP request has been received (or a response has been sent.) This will render the backend superfluous.

My idea was to separate users handlers from the associated connections. Handlers would only interact with request and response object, which, in turn, would interact (by dispatching) with the abstract backend class. About the active or passive party style, it'll lie on the "gluing" mechanism, which is still undecided. Think about sockets configured through an acceptor, for instance. You don't want to go through the acceptor to send a message with/to the socket. About the same happens within a HTTP session. You don't want to go through the backend to respond to a new message. Your callback will be called with the request and response objects and you only want to use the response object to respond to the request. I'll ask for a re-review of this part once the design is more clear and documented. And again, thanks for the detailed feedback. I'll go back to work on the document proposal. And maybe of your interest, I've spent some time playing (and testing) with some ASIO examples to better understand its behaviour under some conditions (eg. send message with an unconnected socket, lifetime management with slightly complex scenarios, simple ordering, ...). This effort will help me to document a design closer to ASIO proposal. In fact, the will verb in the previous sentence is wrong, because the invested effort already **is** helping me with the proposal. -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Vinícius dos Santos Oliveira

17 Mar 17 Mar

6:05 a.m.

Em Sex, 2014-03-07 às 22:54 -0300, Vinícius dos Santos Oliveira escreveu:

...

I don't want to spam the inbox of you guys, then I'll only send this email regarding updates to the proposal and one more once I think it's complete.

As promised, here I'm to inform that I feel the proposal is ready for review, but I continue to appreciate feedback at anytime: https://github.com/vinipsmaker/gsoc2014-boost Or through the ugly formatting at the melange page (with extra information about my skills, not only the proposal per si): https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2014/... -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

Vinícius dos Santos Oliveira

12:54 p.m.

2014-03-17 3:05 GMT-03:00 Vinícius dos Santos Oliveira < vini.ipsmaker@gmail.com>:

...

Or through the ugly formatting at the melange page (with extra information about my skills, not only the proposal per si): https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2014/...

Sorry, wrong melange url. This is the right one: http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/vinipsmak... -- Vinícius dos Santos Oliveira https://about.me/vinipsmaker

3911

Age (days ago)

3927

Last active (days ago)

List overview

Download

23 comments

7 participants

participants (7)

Bjorn Reese
Cory Nelson
Dean Michael Berris
isundill
Jens Weller
Niall Douglas
Vinícius dos Santos Oliveira