Automating Research Team Management Using Experiment Design

Problem: Organizing an Inexperienced Team

In my final year at Georgia Tech, I had three research assistants who had the potential to be productive laboratory technicians, but they lacked experience and confidence. I needed to create an environment where they could hit the ground running gathering high-quality experimental data, documenting their work, and organizing their data without too much micromanagement. Meanwhile, I had publications and a dissertation to write and needed to ensure the quality of my assistant’s work without spending too much time looking over their shoulders.

Solution: Data Management as a Means to Automate Team Managment

A data management strategy can do more than just make data easier to navigate. A strategy which incorporates automated progress updates allows the team members to focus on their delegated roles rather than administrative tasks. Technicians are freed to focus more on their laboratory experiments, analysts can be updated the instant new data is added, and team meetings become more efficient and focused on the next step of the project.

So I began building a data management strategy based on the needs of our project. I would:

  1. Design a set of experiments to explore a parameter space.
  2. Create a data organization system based on that parameter space.
  3. Build tools to automatically track data placed into the system, eliminating the need for my assistants to report their progress.
  4. Expand upon those tools to automate data analysis.

I used Design of Experiments (DOE) principles as the basis of our experiment design and my data management strategy. I started by writing a Python script which built a directory tree which organized according to the parameters of each experiment. Each level of the directory tree corresponded to one parameter from the experiment design.

For example: I developed this data management strategy for a study of fracture toughness in various thin-sheet metallic specimens. Thus, the parameter space had three axes: material, thickness, and type of crack. Thus, the directory structure for the project had 3 layers. For example, an experiment on a copper sheet 100 µm thick with a single edge notch would be stored in the directory .\Cu\B100um\SE.

The ordering of the parameters / directory layers was arbitrary, and I simply chose an arrangement that made it easy for my assistants to find where to put the result of their experiments. For the purpose of organizing our data, the directory tree offered an intuitive, human-readable structure. This structure also made tracking progress straightforward.

Once we started collecting data, I wrote a directory crawler script automatically generated reports on my team’s progress based on the data they collected. This eliminated the need to interrupt my assistants’ laboratory work for progress updates and allowed me to observe our large experimental dataset taking shape in real-time and plan our next move. Finally, I augmented the directory crawler to retrieve and analyze experimental data on-demand.

Flowchart showing how I used software tools to streamline the team management process.

Once the directory crawler could retrieve data, it was straightforward to adapt the tool to analyses. For analytical purposes, it was not always advantageous to view the data in the way it was organized in the directory tree. So I added the ability to filter the data retrieved by the directory crawler based on any arbitrary criteria.

For example, consider our project exploring different materials, sheet thicknesses, and notch types. I could retrieve and compare data from any combination of the three parameters we explored. I could look at all of the single edge-notched experiments on all materials and all thicknesses. Or I could narrow my focus to only single-edge notched specimens from aluminum sheets. All of the data retrieval and visualization was automatic, and I merely needed to identify the scope I wished to examine.

Impact: An Efficient Research Team

Developing my automated team management strategy catalyzed my research team’s progress to such a degree that we accomplished more in one summer than in the previous two or three years. Tasks, like getting a progress report from my assistants, developing a work plan for the week, or applying a new analysis to old data, took only a few minutes, rather than hours of work and meetings. My management framework eliminated my need to micromanage my team, helped my assistants see how their work fit into the larger project and allowed everybody to focus on the science rather than the minutia of organizing the group’s efforts.

My new tools helped me created an optimal environment for my assistants where they always had clear goals, instant feedback on their progress, and a clear record of how their work contributed to the project. At the beginning of every week, I used my directory crawler scripts to compile a big-picture report of the team’s data, identify target areas for the coming week, and set goals for my assistants. I could check on my team’s progress at any time over the course of the week without needing to interrupt their work and then deliver praise or constructive feedback at lunch or the end of the day. This view of my assistant’s daily work even helped me with “sanity management” – making sure my assistants’ days had enough variety and challenges to keep them engaged.

The new management strategy created a highly efficient data collection and analysis pipeline which let me stop worrying about how to collect enough data for the project and shift my focus on developing new theories and models. I had built an efficient data pipeline, and the challenge of analyzing my pipeline’s output sparked my interest in data science. In one summer, my undergraduates collected more high-quality data than some grad students collect in their entire dissertation research. The dataset is so thoroughly documented and well-organized that more than a year later, my old research group is still mining the data for new insights.

Modern data acquisition technology has lowered the barrier to collecting data to such a degree that we can afford to collect data from experiments without every measurement necessarily being aimed at testing a particular hypothesis. Our success that summer hinged on a data management strategy which had the experiment design baked in. By exploring our parameter space in a systematic, disciplined way, we created a dataset conducive to developing and testing new hypotheses and models based on the comprehensive data we had already collected.

Design of experiments allowed us to avoid the traditional hypothesis – experiment – refinement – hypothesis … cycle of the scientific method, where progress is limited by the rate at which experiments can be run. Instead, we explored a large parameter space, and our physical science project became a data science problem. Our progress was only limited by how quickly and insightfully we could develop new models to understand our data.

Postscript: Experiment Specifics

I have kept this post focused on my research team management strategy with minimal discussion of actual research we were working on. I left those details out to avoid complicating the narrative. But for the curious, I thought I would write out some notes on the parameter space we were exploring.

The Goal

Develop a new fracture toughness parameter for thin, ductile metal sheets where traditional analyses such as K and J cannot be applied.

Experiment Parameter Space

  1. Sheet material
    • Al
    • Cu
    • Sn
  2. Sheet thickness (varied depending on material)
  3. Starting notch type
    • Single edge notch
    • Middle notch
    • No notch (tensile)

Raw Data Acquired

  • Force
  • Displacement
  • Optical micrograph sequences (videos of the experiment)
  • Fractographs (micrographs of the fracture surfaces)

Analyses

The analyses we developed were targeted at characterizing the driving force and resistance to crack growth in the ductile sheets. The goal was to find analytical approaches which were insensitive to certain parameters, such as the starting notch type. (I.e. we could use forensic fractography to identify certain parameters as good negative controls.)

But the real beauty of our approach is that we were able to run multiple hypothesis – antithesis – synthesis cycles without the rate-limiting step of waiting to complete more physical experiments. The dataset was large and robust enough that we could simply test different analyses over many different scopes – from single experiments to a comprehensive analysis of the entire set and everything in between. I suspect that there may be a few more Ph.D. dissertations worth of insight still waiting to be discovered in our dataset.

Here are some examples of analyses I developed. The parenthetical statements list the raw data for each analysis.

  • Crack length (micrographs)
  • Stress (force & micrographs)
  • Crack propagation stress evolution (force & micrographs)
  • ANOVA of crack propagation stress convergence (force, micrographs & set groupings based on parameter space)
  • Strain distributions (digital image correlation deformation mapping)
  • Work of fracture (force, displacement, & crack length)
  • Deformation energy distribution (force & deformation maps)
  • Specific deformation energy accumulation rate (force, deformation maps, and crack length)

Measuring Color Part 3: The Observer Functions

Now that we’ve laid out the problem of taking a color measurement from a continuous spectrum, we can actually execute the conversion. We can start by considering the wavelengths of light that correspond to certain colors. For example, red light has a wavelength of about 650 nm, green light is about 550 nm, and blue light is down around 425 nm. In the actual world, light can exist at any wavelength. Color is more complex than just measuring the intensity of light at one specific wavelength.

An example of a continuous spectrum collected from some red-dyed diesel fuel. We want to convert this kind of data into a 3-channel RGB color representation.

The Observer Functions

The human eye perceives red, green, and blue light each over a range of wavelengths. The sensitivity of the eye also varies depending on wavelength. For our project, we will apply some mathematical equations called the “Observer Functions” which approximate the sensitivity of human eyes at each wavelength.

The observer functions approximate how the eye responds to red, green, and blue light. The CIE C Standard approximates the standard light source required for an ASTM D1500 color measurement test. This light source appears white because it stimulates all 3 areas of sensitivity in the human eye.

The observer functions show a few interesting things. First, the human eye is far more sensitive to blue light than green or red, but that sensitivity is over a narrower range of wavelengths. Perception of “red” light mostly covers the longer visible wavelengths. However, the red observer function has a secondary peak at about the same wavelength range as blue light. This suggests that the light we think of as “blue” also stimulates human “red” receptors, albeit not as much as the longer wavelengths we associate with red light.

So how do we use these observer functions? Well, we simply need to multiply each of the observer functions by the continuous transmission spectrum that came from our UV/Vis spectrophotometer and compare that result to the original observer function. Thus if a specimen transmitted 100% of light in the “red” range of wavelengths, the observer function would remain unchanged, and we would see that the red receptors in our “eye” would be 100% stimulated.

Comparison of the ideal observer and standard light source functions (left) compared to an actual spectrum applied to the observer functions (right). Notice how the observer functions on the right are reduced in magnitude according to where they match up to the spectrum.

In the example above, we can see an analysis of a typical red-dyed diesel specimen. The grey curve on the right is the spectrum measured from the specimen, and we can see how the red, green, and blue observer functions are changed by the spectrum. The blue spectrum is practically non-existent. The red spectrum came through at nearly full strength. Finally, the green spectrum was reduced in intensity but has not been suppressed entirely. If we quantify how strongly each range of receptors in our “eye” was stimulated, we could measure the (red, green, blue) color as (0.649, 0.345, 0.004). Mathematically, we accomplish this by summing up the observer functions on the right and comparing them to the observer functions on the left.

So this gives us a color measurement! All that remains is to make some adjustments to make sure our measurement accounts for the conditions of an actual experiment.

Simulating a Standard Experiment

Light Source

While our color measurement does represent the color of the actual specimen, there is an important difference between the UV/Vis spectrophotometer and the standard method: the light source! If we were to follow the standard method, we would use a light source which produces different intensities of light at different wavelengths. The UV/Vis automatically corrects for the light source intensity, so the transmission measurements it provides are as if they were collected using a light source which was equally intense at every wavelength.

For our experiment, correcting for the light source is fairly simple. We just take the spectrum of the standard light source (the “CIE C source” in the plots above) and apply it to our transmission spectrum before we perform our calculations using the observer functions.

Attenuation

There is another difference between our UV/Vis experiment and the standard method: the standard uses a 33 mm thick observation vial and the UV/Vis spectrophotometer uses 10 mm wide cuvettes. So our measurements traveled through less of the specimen than the standard mandates and absorbed less light in total. We can compensate for this using Beer’s Law.

$$A = \epsilon c b$$

Beer’s law states that absorbance, A, scales proportionally with the specimen thickness b. (The other constants are related to the concentration, c, and molar absorptivity, ε, of the specimen, which do not change.) So we know that to simulate the standard measurement, we simply need to scale the absorbance by 3.3 to get an equivalent result! However, our measurement is not in absorbance, it is in transmittance. Fortunately, the conversion between absorbance, A, and transmittance, T, is straightforward.

$$A = log_{10}\left(\frac{1}{T}\right) \\
T = 10^{-A} $$

So dealing with the difference in vial thicknesses is simply a matter of converting to absorbance, applying Beer’s Law to find the equivalent absorbance of a 33 mm vial from the 10 mm cuvette, then converting back to transmittance. Then we will have simulated the standard experiment accounting for both the light source and the thickness of the specimen vial.

Conclusions

I was able to build a color-measurement tool which converted from a continuous spectrum to 3-channel RGB color, which in turn let me match the color to standardized references. Building an automated alternative to the human eye-based standard method required a multi-step conversion.

  • Use of the observer functions to convert a continuous spectrum to 3-channel color.
  • Simulation of the standard light source called for in the standard method.
  • Correction for the smaller-thickness specimen which we used compared to the standard.

To me, the most interesting aspect of this project was understanding how the human eye perceives color. Seeing how the observer functions work showed how in a universe where light can have any wavelength, it is possible to simulate images using only 3 color channels. The usefulness of 3-color images is a result of just which wavelengths of light the human eye is sensitive to.

It is fascinating what this implies. Images displayed on 3-color monitors, printout, and even paintings might look realistic to humans, but any animal with different color sensitivity would likely see them very differently. Representing an image collected by a camera using UV, infrared, X-ray, or other non-visible light could also be accomplished simply by applying a different, specially-designed set of observer functions to convert to 3-channel RGB color. The human eye’s sensitivity is also wavelength-dependent. So a red and blue light might have equal intensities in reality, but a human eye would see the blue light as being brighter.

This all makes me appreciate the human body a little bit more, too. Just “eyeballing” a color measurement seemed like a pretty shaky foundation to build a standardized test on. Using an instrument still gives us a more detailed, objective, and quantitative record of the measurement. However, the eye is actually a rather complex and sophisticated device!

Measuring Color Part 2: The Visible Spectrum vs 3-Channel RGB Color

In my previous post, I laid out our overall goal: use a UV/Vis spectrophotometer to judge the color of fuel specimens. This post will introduce the two different kinds of information we will need to understand in order to solve the problem: the continuous spectrum measured by the UV/Vis spectrophotometer and the 3-channel RGB color value we want to use to identify the color.

The Visible Light Spectrum

We will gloss over the less interesting challenges of repairing the UV/Vis and getting it to talk to a computer. In short, the device just needed some adjustments to its light sources, and it had an RS-232 serial port which I could use to transmit data through an adapter to a listening computer.

The most important piece of the puzzle is the kind of data our instrument produces. To use the spectrophotometer, we place a 1 cm wide cuvette containing our specimen into the sample holder. Then the instrument measures how much light at different wavelengths is transmitted through the cuvette and specimen.

The white-capped cuvette in the center of this image contains our specimen.

So suppose we take a sample of fuel, pipette it into a cuvette, and take a quick spectrum measurement. This is what the resulting data looks like.

Transmittance spectrum collected from a sample of red-dyed diesel fuel.

What we get is a spectrum. The vertical axis shows the transmittance, which ranges from 0.0 to 1.0. A value of 1.0 would mean that 100% of the light was transmitted. A value of 0.0 would mean that absolutely no light was able to get through the specimen. This spectrum was collected within the range of wavelengths that humans can perceive, which runs from roughly 400 nm (violet) to 700 nm (deep red) wavelength. (If you don’t know what I mean by wavelength, you might want to read up a bit here.)

This specimen appeared red to the human eye. The light was mostly transmitted in 650-700 nm range of wavelength, which would correspond to red and orange colors, which makes sense given how the specimen looked before I scanned it. Practically no light was transmitted below 550 nm wavelength, which means the specimen blocked blue and violet light. Somewhere in these ranges of transmitted or blocked light, we will find the information we need to categorize the color of this specimen.

3-Channel RGB Color

But how does this spectrum compare to RGB color? The idea behind RGB color is that 3 numbers – one each for the red, green, and blue color channels – can be used to describe almost any color visible to the human eye.

A screengrab from GIMP showing how different RGB color values can be selected to reproduce a particular color. It is also worth noting that there are many other frameworks for representing color. This one just happens to be the best match to the standard method.

One thing is immediately apparent upon comparing our example of RGB color (255,36,0) to the spectrum from the UV/Vis spectrophotometer. The spectrum contains a great deal more information!

This brings us to the real challenge of the project: converting a continuous spectrum into RGB color values. To do that, we need to consider how the human eye perceives color, which I will discuss in my next post.

Measuring Color Part 1: Setting Up the Problem

This is the story about how a combination of a simple color experiment and limited resources prompted me to design a new experiment and analysis for the lab where I worked. What started as a simple fuel test turned into a project that taught me a great deal about how humans perceive light and color.

The Problem

Suppose you work in a laboratory, and your clients want you to to grade the color of some fuel specimens. They specify that you are supposed to assign a number to the fuel from 1 to 8 ranging from a pale yellow (1), red (4) or nearly black (8), and various shades and numbers in between. Well, it’s good to have clients, and the test sounds simple enough. Great!

But we can’t just eyeball that measurement, and the standard method calls for equipment you don’t have. The lab has limited resources, and you can’t seem to talk your boss into buying the parts you need or turning the clients down. So what do you do?

Opportunity

The standard methods of getting a color measurement require either visual matching to a reference material using a standard light source or use of a specialized piece of equipment. However, visual matching does not leave a quantitative record of the measurement, and we lacked the resources to acquire a specialized instrument. Furthermore, it would be advantageous to analyze colors that fell outside the narrowly-defined yellow-red-black range.

Maybe there was a way to get a measurement that was not only documentable, calibratable, and reproducible, but arguably better than the standard method of visually comparing the specimen to a piece of reference glass!

I found two key resources that would help me escape my conundrum:

  • The lab had an old, out-of-commission UV/Vis spectrophotometer.
  • ASTM D1500 provides the RGB color values for the different color numbers of specimens observed using a standard light source.
Maybe this old spectrophotometer could find a second life measuring fuel color!

So maybe I could get that spectrophotometer working, convert its output to RGB color values, and match it to the nearest standard color number! Then I would have a quantitative record and the capability to identify colors outside of the standard’s yellow-red-black palette.

The Challenge

So I had a path towards a method that I could feel good putting in front of a client.

  1. Bring the defunct UV/Vis back to life.
  2. Find a way to transfer the UV/Vis spectra to a computer.
  3. Convert a continuous spectrum to RGB space.
  4. Match the RGB coordinates to their nearest standardized color value.

Step 3 is where I will focus this series of blog posts because working on the spectrum → RGB conversion taught me a great deal about human vision and color perception. After all, how does the human eye really perceive the visible spectrum? How is it that we take just red, green, and blue light and still create what appears to be the entire spectrum of visible electromagnetic radiation? How can we convert a continuous spectrum to those 3 color channels?

The next post will get to the heart of the problem: How the continuous visible spectrum relates to the red, green, and blue color channels that humans use in our cameras and displays.