One of the issues I run into with docker is the constant lack of reproducibility. During development I've solved this by using devbox. Devbox doesn't solve this efficiently out of the box with their generated dockerfile. And using nix directly is not for the faint of heart.

I've been working on this problem and found a decent workaround using Devbox and Nixery that actually works.

The Problem

• Docker builds aren't reproducible (based on ubuntu, alpine etc)
• Devbox generated dockerfiles don't take advantage of buildLayeredImage in nix
• It doesn't make sense for the team to learn nix (that's why we use devbox)

The Solution

Basically you can base your Dockerfile around Nixery using the devbox.lock commits for each package. This gives you the benefits of Nix's buildLayeredImage without having to write Nix code.

Nixery

The Virus Lounge

The devbox.json :

{
    "packages": [
        "elixir@latest",
        "nodejs@latest",
        "postgresql@latest",
        "ffmpeg@latest"
    ]
}

Then generate a Dockerfile that looks like this:

# Stage 1: Start with shell base
FROM nixery.dev/shell:076e8c6678d8c54204abcb4b1b14c366835a58bb AS base

# Stage 2: Elixir
FROM nixery.dev/elixir:7cc0bff31a3a705d3ac4fdceb030a17239412210 AS elixir-layer

# Stage 3: Node.js
FROM nixery.dev/nodejs:076e8c6678d8c54204abcb4b1b14c366835a58bb AS nodejs-layer

# Final stage: Use base and add packages
FROM base

# Copy nix store from each package stage
COPY --from=elixir-layer /nix /nix
COPY --from=nodejs-layer /nix /nix

# Create a profile directory and symlink all packages
RUN mkdir -p /nix/var/nix/profiles/default/bin && \
    for store_path in /nix/store/*; do \
        if [ -d "$store_path/bin" ]; then \
            for binary in "$store_path/bin"/*; do \
                [ -f "$binary" ] && ln -sf "$binary" "/nix/var/nix/profiles/default/bin/$(basename "$binary")" || true; \
            done; \
        fi; \
    done

ENV PATH=/nix/var/nix/profiles/default/bin:/bin:/usr/bin
WORKDIR /app

The magic is in those commit hashes. They come from your devbox.lock file, so every build uses the exact same package versions.

Why This Works

• Those commit hashes mean identical packages every time. No more "it worked on my machine".
• Docker can cache each package stage independently.
• The packages are layered efficiently by Nixery
• No Nix knowledge required: You just write normal devbox.json files. The script below handles the Nix complexity.

The Script

I made a shell script that does all this automatically. It reads your devbox.json and devbox.lock, extracts the commit hashes, and generates the Dockerfile for you.

A shell script that generates a dockerfile from a devbox.json and devbox.lock using nixery.dev

A shell script that generates a dockerfile from a devbox.json and devbox.lock using nixery.dev - devbox-generate-dockerfile.sh

Gist262588213843476

The script is pretty straightforward and AI generated - it just parses the JSON files with jq and templates out the multi-stage Dockerfile. From there you modify it like any other Dockerfile.

Production Notes

You'd probably want to host your own Nixery instance instead of using the public nixery.dev. The concept is the same though.

This gives you the benefits of Nix's deterministic builds without having to learn Nix. Your Docker images are smaller, your builds are reproducible, and you don't have to deal with package drift anymore.

The current docker generate in Devbox works but isn't as optimal as this approach. This gets you closer to what buildLayeredImage does natively in Nix, just wrapped in a more familiar Docker workflow.

If you need help deploying, our team at Lambdo is ready to help.

If you're not familiar with NixOS the basic pitch is:

Declarative builds and deployments.
if a package works on one machine, it will also work on another.
trivial to share development and build environments
installing or upgrading one package cannot break other packages

I've been running NixOS on my personal machines and in local and cloud servers for almost a year now. I started playing around with it locally initially and eventually moved it to production uses. It definitely has a somewhat steep learning curve, but in my case it was enough to start with an example and build it up from there.

The official docs and package search is functional enough but leaves a bit to be desired in detail. But those are getting better. In my case these resources were indispensable:

GitHub - Misterio77/nix-starter-configs: Simple and documented config templates to help you get started with NixOS + home-manager + flakes. All the boilerplate you need!

Simple and documented config templates to help you get started with NixOS + home-manager + flakes. All the boilerplate you need! - Misterio77/nix-starter-configs

GitHubMisterio77

Vimjoyer

Teile deine Videos mit Freunden, Verwandten oder der ganzen Welt

YouTube

First steps — nix.dev documentation

nix.devNix documentation team

It's nice for personal use because tinkering with a linux setup, packages, desktop environment, os config is really low risk. It's easy to roll back to a previous version, and the whole system and user config (through home-manager) can be kept in a single git repository.

I've been using NixOS personally on old laptops, handheld computers and raspberry pis. The first configuration was the hardest. It took some time but with the resources above and digging through the nixos forums I was able to get the first draft running on an old macbook air.

If anyone is serious about learning NixOS for personal use I really recommend installing it on an old computer that's sitting around or to boot it up on a virtual machine.

Using NixOS in production so far has been a great experience for us overall. First off, a bit about my team to explain how we use it. We are a small team of developers that for the most part work on a contract and retainer basis and offer some software products. We build apps, services, integrations, and AI products for our clients. We specialize in rescue projects, meaning clients come from issues with previous tech or developers. We also manage hosting and cloud resources for our clients including NixOS servers. If any of the above is interesting feel free to reach out 🙂

So we have multiple use cases for NixOS. We we use it to self host internal services, we use it to host our other product services, we use it to manage our clients services on their cloud providers, we use it in proxmox vms, we use it for development at the repository level through devbox. I cover the basics below, but I might cover the actual implementation of each use case in more detail in a future post.

In our case it was very easy to take my personal nixos config and and adapt it to a production server. Obviously there are differences between OS setup for a local device and a vm server but those differences aren't too much of a concern. For example one cloud provider that we use is Hetzner. They don't provide a NixOS image by default, but that's easy to overcome with NixOS-Infect. NixOS-Infect installs the operating system onto a vm in Hetzner and generates a configuration file. That configuration file is already valid and can be saved in a repository. Any settings and configuration from a personal config can be copied over, stored in the same repository, or can be shared by creating a nix module.

Hosting our services

This post you're reading is actually using the open source Ghost Blog on a NixOS server on Hetzner! Similarly, we host other open sources services that we use internally. Some services use docker images through nixos oci-containers. Others are just nixos packages configured to run as systemd services. Some of our product services are also running on nixos. One of the big benefits for using nixos in production for us is that all of the configuration is very explicit and can be seen in the configuration repository. Meaning we won't forget if we make a configuration change to the server directly because it's will have to go through the repository first. It also makes moving projects between cloud providers easier.

Proxmox NixOS VMs

Github codespaces seems like a great product to provide powerful machines to thin clients. We use nixos vms on proxmox as a self hosted alternative. Each vm can be configured for each user, the same configuration can be applied and deployed to all vms at once. We can even add code server to each vm so that they can be accessed from a browser!

This was actually our first usage of nixos in our business. It was not all that different from running on an actual device. The differences were actually in the specifics of using proxmox.

Devbox for each repository

We also use Devbox for each project that we are working on to create dev environments. Devbox actually uses nix under the hood. Devbox removes the need to manage versions of packages manually. For example we have used n and volta in the past to manage versions of node, that's not needed any more. We can manage versions of nodejs, yarn, and any other external dependency all through devbox. All developers have access to the exact same dev environment which removes most instances of "it works on my machine". Similarly this devbox configuration can be used in production as well with a docker build or as a systemd service.

Devbox is a great introduction to getting the benefits of NixOS, without having to go through the steep learning curve.

Overall using NixOS has been a great experience. It has made our production services more explicit and our local builds reproducible. I'd recommend it to business or developers interested in declarative and reproducible builds. If you're interested and have any questions, feel free to reach out and I'll point you in the right direction.

For the last few months I've been eagerly working on building a Natural Language Processing model. I present  Semantic Parsing English-to-GraphQL (SPEGQL).

In this post, I'll cover some of the highlights, and include my presentation at OpenAI.

Background

GraphQL

GraphQL is a query language for your API.

Recently GraphQL has gained a lot of popularity. Some of the main benefits I'll note are: it represents the API's schema as a graph, nested relations of any depth can be easily queried, it can aggregate data over multiple datasources and responses are predictable among other things.

Semantic Parsing

Semantic parsing is the task of converting a natural language utterance to a logical form: a machine-understandable representation of its meaning. In this case I wanted to semantically parse from English, to GraphQL.

Why this project?

I had a few reasons to work on this project:

• I wanted to understand the limits of general language models for Semantic Parsing
• This project could potentially ease the learning curve for new developers of GraphQL
• It has potential use as tooling for non technical data users, such as managers, to gain insights into their data

Objective

Given an English prompt:

“What is the name and date of the song released most recently?”

And some GraphQL Schema

type song {
 artist: artist
 artist_name: String
 country: String
 f_id: Int
 file: files
 genre: genre
 genre_is: String
 languages: String
 rating: Int
 releasedate: String
 resolution: Int
 song_name: String
}

...

Find a corresponding GraphQL Query:

query {
 song(limit: 1, order_by: {releasedate: desc}) {
   song_name
   releasedate
 }
}

This objective could be tested by passing the prompt and schema though a model (In this case T5) to output a query. The process is as follows:

Methods

The process required multiple steps

1. Create an English to GraphQL dataset
2. Run experiments on Encoder-Decoder Transformer models (Bart and T5)
3. Collect data and results
4. Implement a graphical interface to interact with the model

Results

• 46 - 50% exact set matching accuracy on GraphQL validation dataset

A couple of example videos will help show results as well

Other Notes

Here is the main Repo for creating and validating the Dataset:

https://github.com/acarrera94/sql-to-graphql

I also created an example notebook for anyone who wants to try out the model. This model is finetuned on GraphQL and SQL and can create queries for both languages:

https://colab.research.google.com/drive/1l1h8RlEl-IS0XfkDh66qikH4UsD19KF6?usp=sharing

I'm currently working on a paper that details the whole process, that's linked here.

And finally, I also gave a presentation about my project at OpenAI:

OpenAI Scholars Spring 2020: Final Projects

Our third class of OpenAI Scholars [/blog/openai-scholars-spring-2020/] presented their final projects at virtual Demo Day, showcasing their researchresults from over the past five months. These projects investigated problemssuch as analyzing how GPT-2 represents grammar, measuring the interpreta…

OpenAIOpenAI

The scholars program is coming to an end. Since my next post will detail my final project, I thought this would be a good opportunity to reflect on the scholars program as a whole. I'd like to talk about the goals of the scholars program.

Goal of the Scholars Program

The goal of the Scholars program is to improve diversity in the field at large. This was worked towards by providing stipends and mentorship to individuals from underrepresented groups to study deep learning and produce an open source project.

Self Motivation

Since the objective of a scholar in the program is to study deep learning and produce an open source project, a scholar should be very self motivated and technically proficient. Self motivation is important when developing your own curriculum/project as is done in this program. And the technical proficiency is a good signal that a scholar will be comfortable creating an open source project.

The program was set up in such a way that it was flexible for each scholar to learn/work at their own pace and work on their own domain/project. I personally found that flexibility awesome. I think of myself as very self motivated and I believe that, that was to my benefit during the program. I spent the first part of the program learning about the state of the art in natural language processing and contributing to open source. And I spent the last part of the program working on my final project. I think that to thrive in this program, you must be very self motivated and willing to learn.

In my opinion, this cohort of the scholars program is was very technically proficient. So I believe that any of the scholars in this cohort will be able to contribute to the field. Although our level of experience with respect to deep learning varied from little to none, the experience with the technical definitely helped everyone build a foundation for future deep learning work.

Assuming the scholars program continues to have similar expectations, for someone to be a successful scholar in the future, self motivation and a high level of technical proficiency will be essential. Anyone reading this article looking to apply for a future cohort of the Scholars Program (or a similar program) should consider those skills. Since this is only the third cohort, the scholars program might change in the future, so it's helpful to keep an eye out for the current requirements and expectations in the scholars program. Recently, OpenAI announced that the scholars program will have another cohort this fall. For more on that follow the OpenAI blog.

Diversity

Since the start of the program I have thought about that main goal to improve diversity in the field and what that means for me, and I have some thoughts to share. Although conversations about diversity are interesting, helpful, and necessary, unfortunately issues such as these are also very "complex, sensitive, and nuanced". I believe that these conversations deserve to be talked about in depth, acknowledging the underlying assumptions and nuances, but I don't feel I can do the whole topic justice in a short blog post format. For that reason I'll try to keep it simple and as specific to this program (or deep learning) as possible.

The goal of diversity programs such as these are definitely noble, I would love to see more people from many backgrounds participate in deep learning specifically. I believe that diversity of thought/background is helpful in the creative process, specifically in divergent thinking. And in a field such as deep learning where there is much to be discovered, divergent thinking is a necessity. And it was awesome to me that the program itself was very flexible, allowing us with varied levels of experience to grow in deep learning. That's why overall, I think that the scholars program is a great way to introduce that, I applaud the effort and I believe we'll see great results from this cohort soon enough.

Gratitude

I'd like to give thanks to some communities and people that have been helpful throughout the

OpenAI: For the opportunity to be part of this program, connect with experts and setting up some future opportunities.

Huggingface, Pytorch Lightning: Very helpful in sharing resources and answering questions.

Melanie Subbiah: my mentor for the scholars program, always insightful and helpful. Good mentorship is rare but Melanie proved to be a wonderful mentor for me.

My wife Noelle for being a support throughout this journey!

My final project involves taking natural language English prompts and running them through a model to generate corresponding GraphQL queries. This post details the preliminary model experiments and validation strategies I've used the past couple of weeks.

Original Plan

As you can see in my project proposal document, I planned to experiment with a few models including the BART, the vanilla transformer, and the Reformer. I've stayed close to that original plan though I replaced the vanilla transformer with T5. The idea was to run some preliminary experiments with each of those models and see collect results that later on, I would be able to improve these metrics.

Roadblocks

I think it's helpful to reflect on some of the issues I found as I started training the models. There were some minor issues including errors with my dataset, hyperparameter issues, and computation constraints. But those roadblocks were easily fixed with a little debugging. The biggest issue I ran into that I didn't plan for was my evaluation metrics.

Initially, I just assumed I would be able to use the same evaluation metrics as Spider (the dataset my dataset was based on). Then while running through my models I realized I needed to do a little more research. Eventually I landed on the idea of taking the results from my model and my target queries and parsing them into an Abstract Syntax Tree. I could then use set comparison between the ASTs and get a boolean accuracy for each example. Luckily I found a library to parse GraphQL queries into an AST. I had to make a few minor tweaks but eventually I had an accuracy validation metric I could run my models against.

Results

Once I validated my models on my validation accuracy, I was pretty excited with my results. I got 48% accuracy on my T5 model! It seemed like a big deal to me since this is a new dataset and a new task and I'm already getting results without much hyperparameter turning.

Future Work

The validation metric that I came up with seems to work for my purposes but there could be improvements. Since queries are just trees based on the schema graph, there are multiple ways to represent equivalent queries. So ideally I could come up with a way to evaluate those queries equivalently, though I'm not sure my time constraints will allow it. Other options include adding execution accuracy, to verify how queries run against my GraphQL endpoints.

In the coming weeks I'll continue to experiment with my models, collect further results and compile everything I've gathered from the program!

My final project for OpenAI is to generate GraphQL queries from natural english prompts. The first project deliverable is to create a GraphQL dataset. This post will detail the process I went through to create that dataset and my reflection of each step.

Research & Plan

I started doing some preliminary research as I prepared my project proposal. This included researching different tools and resources that I could use to create a dataset. The tools I used are detailed in my project proposal. It took a bit of balancing the available tools with the tools I'm comfortable with. One of the most important resources I used was the Spider Dataset, a Text-to-SQL dataset. My idea was to build off that dataset and convert it to GraphQL. It took some research to even realize if it was even possible. I spent days looking at SQL and GraphQL grammar and I've reviewed the dataset throughout. I took notes about possible approaches, went through different exercises until I felt comfortable with an approach. It took around 2 weeks of research and planning before I felt I had a solid approach and the right tools.

I felt that those 2 weeks were enough to have a solid foundation to build on, maybe a little more would have been helpful. Too much more time and I would have lost time running experiments. Any less and I might not have enough underlying understanding to build the dataset.

Coding & Building

Building the dataset took a decent amount of engineering. All in all it, I wrote 7 scripts and around 2,000 lines of code, plus some more generated code. Most of the process went pretty smoothly.  I don't want to go into too much technical detail but there were some choices that ended up being instrumental in building the dataset.

One strategy I took was to use the language and IDE to offload cognitive load as much as possible, since I expected the engineering to get intense. I used Swift for its type system since I was dealing with structured data. It's also a language I'm comfortable with, and it's speed and debugging tools were also beneficial.

I also found myself making a lot of assumptions about the dataset, so I would validate these assumptions by littering my scripts with assert statements and fatal errors. This had the secondary benefit that I gained a decent understanding of the examples in the dataset.

Surprisingly I only ran into a couple issues in the process, though these are decently technical and specific to my dataset. One of the big features of the Spider dataset was that it includes many complex queries, the issues I ran into were in these queries. For example, one issue was that Hasura (the tool that builds a GraphQL endpoint on top of a DB)  has the limitation that its default endpoint can't represent the GROUP BY clause. With issues like these I had to decide between writing manual queries or throwing out those examples.

I did expect to run into more issues while engineering but I was pretty happy with how smooth it went and how the dataset turned out. This ended up taking around 3 weeks before the dataset was at a point where I could start validating it and running some deep learning experiments with it.

Review & Validation

This step had me going back to the dataset and fixing issues that I found after the fact. This process included manually looking at examples and verifying the queries syntax and ran on Hasura. There were some issues I only ran into while running the dataset through a baseline network. Luckily I found most issues by writing a script to validate the queries with an eslint plugin.

Although this step has added more work than I expected, I learned the obvious lesson that review and validation is necessary to build a high quality dataset.

Next Steps

I'm actually not finished with reviewing my dataset. I'm still catching and fixing minor issues. But I'll be constantly improving and adding examples until the end of the project.

At the end of the project I'll be releasing the dataset along with all of the source code. For now I'll be working on building those baseline models and improving the dataset.

The OpenAI scholars program ends with the deliverable of an open source final project. I'll be working on this project from now until the start of July when the program ends. This post will give a brief overview and a link to my proposal for that project.

Text-to-GraphQL

The core goal of this project is simple. Given a natural english question and a GraphQL schema, give an equivalent GraphQL query. This means that the question and the schema will pass through a model, and that model will return a query that represents the question correctly.

For example the question:

How many players do we have?

Will return the following query:

query {
  players_aggregate {
    aggregate {
      count
    }
  }
}

The query could be sent to the GraphQL endpoint and return a valid result.

There are a few deliverables leading up to, and following this main objective.

Mainly:

Dataset creation (building a GraphQL dataset based on the Spider SQL dataset)

Training Multiple models (BART, Transformer, T5, Reformer)

Interaction tool (input questions into a GUI)

Submitting a Paper (including info on my dataset and and results from models)

The full proposal is found here:

OpenAI final project proposal

Text-to-GraphQL Andre Carrera April 6, 2020 Description Over the last several years, there has been a lot of headway made on text-to-SQL tasks through the introduction of new datasets and challenges. The problem presented is, given a natural english question and a database schema, produce a SQL q...

Google Docs

I'd love to hear feedback! Please reach out to me on twitter @AndreCarreraPaz

Motivations

So why this project?

The main reasons I'll explain below are that:

1. This is an intersection of my interests. NLP, GraphQL. Deep learning and Engineering.
2. I think this will be a useful contribution to the community.

NLP is a field that has advanced significantly in the last few years. With GPT-2 for example, we can now see a network generate text based off of any prompt. It's fun to play with, and there's even games that have come from it. BERT, is another NLP tool that can help with things such Text Classification and Categorization and even question answering.

GraphQL is an amazing developer tool. It's a query language for your API. That means that you can have a typed schema on top of any API with any set of business logic and databases underneath. It's nice that data is returned in a hierarchy and the tooling that has come with it is incredible. For example, there is GUI schema exploration with GraphiQL. And The Apollo libraries can generate typed code for your frontends.

Also, I like the idea of combining expert systems with deep learning models to get the best of both worlds. Broadly, this means that can use an unstructured question to get results from structured data. This combined with the ease of learning GraphQL makes for an interesting project. And hopefully with the release of dataset and code accompanying this project, future work will improve upon my results.

Stay tuned, the following blog posts will cover progress on my final project.

These last couple of weeks have involved a lot of code, from my PR to huggingface to my PR to Swift for Tensorflow, to my current work on implementing  a text-to-SQL model on the Spider dataset. Since that time was spent mostly coding, I feel like I have picked up on a few differences between deep learning and engineering.

As I've mentioned in previous posts, I come from a software engineering background. I've deployed client projects in Vue.js, Scala, SwiftUI, Kotlin, and Node.js. What that means is that I have a decent amount of experience in both frontend and backend development. I realized that deep learning definitely has a different feel and workflow than fullstack development. What follows are some of the differences I've noticed as a novice in deep learning.

Feedback

The feedback loop is the biggest difference I've notice between DL and engineering. The time spent between writing code and seeing the results can vary dramatically.

For example when I'm writing code to load the dataset, or prepare a dataset, it's very comparable to writing backend code. In both cases I have data I want to take data from one place, process it, and return it to another place. Verifying that code can be pretty straight forward, just run some data through it. It's even possible to write test cases for the dataset.  

Even the first steps of creating a model are similar to engineering. An architecture / design pattern is chosen and you can move forward implementing and connecting the pieces. If you run into an error at run time, it's not incredibly hard to debug, since you usually know where to look based off of a stack trace.

Where it gets rough is when you start training the model on the data. This is not comparable to any other type of programming I've experience before. Your model could output NaNs, after a while of training and it takes some time to understand where that came from, your model could output a negative loss (you probably want positive) and you don't know what's causing it. Maybe your model doesn't learn; where do you even start to debug that? Finding these sorts of issues, at least for me, are very time consuming and it seems like it can only be solved through a lot of trial and error.

Then, the feedback loop can get really long. At this point, it's a matter of waiting for the model to train and checking the results. This process can take hours depending on the kind of data and the dataset. The unfortunate part is that this part is necessary to test different hyperparameters to get the best results.

Though it might seem like a frustrating process, it can be very rewarding. And that's a perfect segway in the next section, rewards.  

Rewards

In engineering the feedback loop can be pretty short, especially in front end development. For example using something like Vue.js, React, or SwiftUI, means that as I'm writing the code for some view, I can see that same view updated in real time on the same screen. This sort of work is immediately satisfying since you can constantly see exactly what you're making. That feedback loop is longer in backend development, since it can take time to see results and therefore more time to feel satisfied with a feature. Then in DL, that feedback loop can be stretched even further.

This doesn't mean that DL is any less satisfying. On the contrary, it can be more satisfying. Just with more varied time between results.

For example, a couple of weeks ago, I was working on training a model for a summarization task. It took me more than a week to get it working, I ran into the issues I described in the previous section. And at times it got a little frustrating. Then, I finally got some results. I summarized a CNN article and got the following results:

CNN's Soledad O'Brien takes a tour of the "forgotten floor," where mentally ill inmates are housed in Miami before trial. An inmate housed on the "forgotten floor.

I was ecstatic to see these results. These word combinations didn't appear together anywhere in the article, but the model was able to create this text which summarized the article and (somewhat) maintained the grammar.

This last week I was working on the text-to-SQL task. Where given a SQL schema and an english question, such as: "How many singers do we have?". Output an equivalent SQL query. After some experimentation, I got the following:

SELECT count(*) FROM singer GROUP BY singer_id

This is a valid SQL query. The model learned SQL grammar and learned to translate english to a SQL query. These are the kinds of results that get me excited about deep learning. Though this process was very time consuming, and there was frustration involved, I felt very satisfied with the results.

I've been getting very excited about my final project. In my next blog post I will describe my final project and share my project proposal.

These past couple of week I've been working on a contribution to the 🤗Huggingface Transformers repo. Transformers is a fast moving library that uses the transformer deep learning architecture to solve problems such as translation, named entity recognition, summarization, and other nlp tasks. I also used the PyTorch Lightning framework. Lightning is a framework used to simplify and organize training and inference code.

This blog post is going to cover my experience building a system and turning that into a contribution to Transformers.

Open Source Contribution

My goal was to use the BART model for summarization. I did have access to a pre-trained model but I wanted to go through the process of training and evaluating it on the full dataset. The basic idea is that given some text article, output a short summary of the article.

There were several issues I ran into along the way. Debugging these sorts of issues while learning a library was really time consuming. I spent hours on simple problems where the solution was only 1 line of code. Understanding the library was very important, I wasn't afraid to dive into the source code, and step through simple examples. Then, once I finally got results with smaller examples I started experimenting with the hyper parameters. I knew that the intuition behind hyper parameters is a trick of the trade. And that's a little frustrating coming from an engineering background. Like I've heard before, it is more of an art than a science. My solution was to run many experiments and see what happens, unfortunately I haven't heard many better alternatives.

After days of tinkering, I got results I was pretty happy with.

Given a CNN article the model generated the following summary:

CNN's Soledad O'Brien takes a tour of the "forgotten floor," where mentally ill inmates are housed in Miami before trial. An inmate housed on the "forgotten floor.

Pretty cool huh?

So why did I decide to spend some of my time contributing to an open source project?

The first motivation was that about a year ago, I talked to the owner of a local AI company. He said that applicants stand out when they've contributed to deep learning projects. It shows self motivation, and that you can learn and contribute to an existing codebase.

The second motivation was, "Why not?". I actually didn't start my project by thinking of how I could contribute to Huggingface.  I started it with a desire to understand how to train a language model for summarization. Then once I realized I built something that wasn't available to other people, I cleaned up the code and turned it into a pull request.

The rest of the pull request was pretty straight forward, I just had to make some small requested changes, and it ended up being a very constructive experience. Although this contribution did take time and I struggled through it at times, I came out with a deeper understanding of the Transformers and PyTorch APIs. I also got invited to some slack channels to discuss this implementation, it turned out to be a great way to build on that community I talked about in my previous post.

In my next post I'll be talking about the differences I've noticed between deep learning and engineering.

While writing this blog post I realized that since the field of deep learning moves so fast, having too many technical details won't be useful as time goes on. So I’ll try to focus more on sharing the details of my learning process and the lessons I've learned.

Lessons learned

First big lesson, community is invaluable.

These past two weeks I have been learning about language models. Many times I found myself with questions and a fuzzy understanding of how things worked. So I would reach out to my mentor, Melanie, and ask questions. I found it really helpful to talk through my confusion and explain those concepts to her and get feedback. I also spent time implementing a CoQA leaderboard paper with another scholar, Pamela. That pair programming helped me see different approaches to problem solving. And since I'm remote, I've been reviewing language model topics with local colleagues.  All of these conversations were essential to building an intuition and a working understanding.

I also spent time going over a huggingface tutorial, and ending up contributing to the repo. The goal of the tutorial was to build a conversational bot using transfer learning for transformer models. Since my final project will be closely tied with transformers, I thought it would be helpful to dive into the code. While I was exploring the code and attempting to fine tune the model, I noticed that the repo was using an old version the library and I tried to update. The update wasn't trivial and I ended up submitting a pull request. I found it really valuable to interact with maintainers and it was satisfying to contribute something as a newcomer.

And finally I have also been attending the Swift for Tensorflow design meetings. It's exciting to be part of a community building a tool that I believe in. And it's very helpful to see what goes into building a tool for deep learning. I'm hoping to contribute more in the following weeks.

Second big lesson, tooling is invaluable.

At first I started doing everything on Google Colab notebooks. The hardware that comes with colab is decently performant and I thought it would be a good way to start and get quick feedback. One problem I had was that I found myself running into a lot of errors, and I instinctively started printing out shapes and tensors to get an idea of what was going on. Eventually I realized how much that process was slowing me down and I downloaded my code and ran it on an IDE with debugging enabled. The IDE was helpful in understanding the flow and architecture of the programs. There are benefits to both tools so I found myself switching between the tools depending on task. I think it's always valuable to know what tools are available and their limitations.

Last big lesson, direction is invaluable

I have found that having a direction to work towards, a goal or a vision, is the most important part of staying motivated. This isn't something specific to deep learning and isn't something I learned these last couple of weeks. But I've always found it helpful in staying motivated in any big endevor, and the Scholars Program isn't any different. The vision I've been working towards, in this case, is my final project. I started writing out a draft for my final project as soon as I knew I was accepted to the program. Since then, all of my work has been building towards that final project, and it has kept me motivated.

Next time I'll be converging even more on resources that will help with my final project. And I'm excited to get more experience under my belt!

My name is Andre Carrera, I am the founder of Lambdo.

For the next 4 months (February - May 2020) I'll be part of the 3rd cohort of the OpenAI Scholars program. I ran into this program by coincidence last year. I took Dr. Wingate's deep learning class my last semester at BYU, and I as I was exploring opportunities in the field, I ran into a blog post for the program. I quickly applied, got through some interviews and here I am.

Every two weeks I will write about my experience. I'll try to be as detailed as possible and hopefully these blog posts will be useful for anyone interested in getting into the field.

Self-Education

I really believe self-education is a legitimate option for anyone interested in Deep Learning, or in development in general. Shorty after starting my university experience in CS, I realized that I wasn't going to be taught skills that would be applicable in the real world. It might just have been my university, but I felt that there was too much theory, too much busy work and not enough application. I only found a few of classes to be useful thus far, and I could fit all of those into a semester.

So, I took measures into my own hands and challenged myself to learn those skills. I did that by taking on software development projects. I took on projects not knowing how to use the technology but being fully confident that I would be able to learn. Lucky for me, that turned out to be the case. I found that having real projects with real goals and real outcomes was extremely helpful for my education. Not only was it good motivation to learn, but I was also learning real marketable skills.

I see the OpenAI scholars program as another opportunity to learn valuable skills. I'm lucky enough to create my own curriculum and work on my own open source project. I'm especially excited about the opportunity to rub shoulders with experts in the field. As part of the program I created a syllabus that I'll update throughout the program attached below.  

Syllabus

Exploration

I decided to use the first two weeks to explore, find interesting papers, learn some of the foundations and get a feel for how fast to approach things. I started out by going through Ian Goodfellow's Deep Learning Book and reading some papers on my syllabus. After a day of that I realized I couldn't passively take in so much information. I'm used to engineering where there is an objective and any reading and learning is quickly applied. So I decided to change my strategy a bit. I'm going spread out the hard technical reading, and focus on more application and implementation.

I spent the week building an understanding of self attention and transformers, since the next few weeks and my final project will depend on that understanding. For anyone one interested in those topics The Illustrated Transformer was extremely valuable.

I also decided that throughout the project I'm going to contribute to Swift for Tensorflow. In fact this week I started working on a pull request to implement an example transformer for translation, it'll be based on the annotated transformer. I'll be posting links once it's done.

Next week I plan to work Conversational AI with Transfer Learning. I'm interested to see what kind of result I can get with transformers and what other baselines I can implement.

If you have any questions, please feel free to reach out to me!

Andre Carrera