Data to Insight is not a software supplier in the traditional sense -- we're here to help local authority (LA) colleagues get work done and share it with each other. The best thing about Data to Insight is the community of analysts who are doing and sharing great work through this network.
We recognise that different LAs want different levels of engagement with our work. As such, we offer four key collaboration approaches to suit different users’ information needs.
Python learning tools
This is the page where you will find our crash course in Python videos, and other learning materials. These are intended to get you from no Python, through to the level where you can contribute to validation rule writing, and start using the dev and usability versions of tools. There are also some beginners Python guides and tutorials. You will also find out Python tools and Notebooks. These come in three versions in usability, dev, and learning versions (eventually, the tools will all be available in each version). The different versions of these tools are explained below.
If you're an absolute Python beginner looking to learn, start with the videos, if you've done a bit of Python and for some reason don't want to try the videos, try the Beginners worksheet. The order of the sections below gives a guide for the order we think absolute beginners should approach them.
​
Some important notes:
-
To follow the links to the code on GitHub, you'll need a GitHub account and membership of the D2I GitHub, once you have an account email Will at william.levack-payne@eastsussex.gov.uk to get membership of our GitHub.
-
To use some tools, you'll need to have Python and a development environment installed on your computer, unless you're using a browser based environment like in the repl.it guide. At D2I we recommend asking your LA to install the standard distribution of Anaconda as it comes with Python, key Python Packages, and a number of development environments, most importantly for our tools, Jupyter Notebooks. Anaconda can be found here: Anaconda | Anaconda Distribution.
-
Not all tools currently have all three of usability, development, and learning versions in a release ready state, the current available versions are indicated in each section.
Tool types:
Usability tools
...are intended for those with little to no Python and Jupyter Experience. The easiest tools to use and get started with, they should get you comfortable with the Jupyter environment, and the kind of things you could do in python relatively easily.
Dev Versions
...have the same outputs as the usability versions of the tools, but without some beginner-friendly features. This makes the code massively simpler, so it's easier for budding Pythonistas to read and change the code, also, in most cases it allows every cell to be run in one go, rather than waiting for user inputs as the inputs are hard coded.
Learning versions
...allow users to start using the skills and tools needed to eventually build their own tools. These are complete tools with lines of code removed, and prompts to help you fill it back in. When finished, the outputs will end up being the same as the completed tools. These are intended for users who want to apply Python experience to building analysis tools.
Worksheets
...are intended for people just starting with Python. You should probably know some things first, like how to navigate Jupyter Notebooks (you can find a guide below), and how to do some basic things like using print(). The first D2I worksheet is a whistle stop tour of the key things you'd need to bring Python into day to day work as an LA analyst
Tools and guides:
01
There are two different resources here. The first is the set of YouTube videos collated to help get people introduced to Python and up to speed for getting involve din the PATCh tool. They are not made by people associated with D2I and are really jumping off points. The second set of videos were created for D2I by Social Finance and cover the basic Python needed to get involved with coding for the CIN and 903 validators.
​
Both of these resources cover much of the same material, but the PATCh resources have a focus on visualisations and analysis, where the validator resources have a focus on using Pandas to make very specific selections necessary for validating data.
​
​
As well as these sets of video resources, check the rest of this page for other key resources such as workbooks and tools to help you get learning. Particularly, if you don't have Python locally, check out the JupyterLite set-up guide.
Available versions: Videos and accompanying documents, this is not a tool so doesn't have versions.
02
D2I is now moving away from Codespaces and towards JupyterLite as our preferred method of Python collaboration, but this guide is still here as it's needed for those getting involved in coding validation rules. For those interested in getting involved in the PATCh tool, see the JupyterLite setup guide.
The guide takes users through the process of getting code from the D2I Github and into the Codespaces coding environment, making changes, and then pushing it back to the D2I Github to be merged by a reviewer. This will be useful for anyone who wants to contribute tot he D2I codebase, be it writing validation rules for our larger tools, or building their own tools and adding them to the repo. Find the Github and Replit guide here.
​
A deprecated version of this guide that covers the same process but for Replit can be found here.
Available versions: Text based guide, this is not a tool so does not feature versions.
03
How to use code a CIN rule on Codespaces:
This guide explains how to code a rule for the CIN validator using GitHub and Codespaces. There is a video guide, a quick text guide for experienced Python coders and to refresh returners, and an in depth guide with detailed text and image descriptions. It also includes some information about troubleshooting. This guide assumes you have completed the Codespaces and GitHub guide above. Differnet rule types, as indicated on GitHub, have their own guides:
Available versions: Videos and accompanying documents, this is not a tool so doesn't have versions.
04
Jupyter Notebooks are a really useful way to both perform data analysis, and also present worksheets for teaching Python filled with lots of examples and explanation, it's not without it's quirks however. This guide teaches the basics of getting it running and using a notebook. Find the Jupyter Notebooks guide here.
05
This worksheet allows you to read pre written code, and follow prompts in text and in-code comments to write your own bit of data analysis using Python in the Jupyter notebooks environment. If you've never used Jupyter notebooks before, it's recommended that you complete our How to use Jupyter Notebooks tutorial first. It is also recommended that you watch and learn along with the Python video tutorials up to and including lesson 3 first of all if you have no Python experience. This worksheet lets you practice key skills for Python data analysis: loading data into Python and reading it in Pandas, selecting interesting bits of data, performing calculations on the data, and visualising that data by plotting it.
Available versions: This worksheet only comes in learning and answers versions.
Available versions: Text based guide, this is not a tool so does not feature versions.
06
The disproportionality tool allows you to analyse and visualise disproportionality, comparing between a user population data, and DfE data for all LAs nationally. Three versions of the tool exist, all running in Jupyter notebooks. There is an ease of use version, a dev version, and a learning version which allows users to write their own tool. Find out more and see the Disproportionality tool here. Note that whilst this tool asks for ethnic group population data to be a functional tool, you do not need to worry about this if you just want to use the tool to learn Python as sample data is pre-loaded.
Example visualisation from the Disproportionality tool, using fake data.
Available versions: Usability, development, learning.
One of the Drift visualisations using fake data: spread of Referral to CLA wait times
07
The Drift tool allows users to input Referral, CP plan and CLA dates to analyse an visualise drift: how long children wait between stages of Children's services provision, and how it changes year on year. Find the Drift tool here. Note that to use this tool you will need to have access to, or request some data sets following whatever procedure your LA uses.
Available versions: Usability, development, learning.
08
The Annex A visualisation tool is intended to demonstrate some of what is possible in Python. It reads from the ChAT tool and then, using the calculations from the ChAT tool, makes these same visualisations again. The idea is not that it does the work of the ChAT tool, as it relies on the ChAT tool, more that it is a quick way to show how we can get nice visualisations out of Python with relatively little code. It is important to note that to run this tool, you will need a version of the ChAT loaded with recent Annex A data.
Available versions: Development
09
On this page you can find explanations of, and links to, the code done in Will's day-to-day Python workshops. This is useful if you couldn't attend and want to see what we did, or even if you just want to copy/paste a bit of code.
10
JupyterLite loads Python into your browser to allow you to write and run Python without having to install it locally, but does so in a way that all the Python is still happening locally, meaning you're safe to use any data you want as it never leaves your computer. This is explained in detail in the guide, and also gives a demo of the kind of things JupyterLite can do, useful if you're interested in using it, and also if you want to show people what's possible and reassure them that it's safe. The link to the demo is available here (it may take some time to load!)
11
This page explains how to set up your own version of JypyterLite to allow the use of Python, entirely in browser, with no need to have it installed locally.