Blog · HKUST Summer Research

Counting Crowds in Hong Kong: What I Built and What I Learned

This summer I taught a computer to count dense crowds of people in video, and spent a good part of it learning how to tell whether a model is really working.

Left: the HKUST campus over Clear Water Bay. Right: me, with the Hong Kong skyline at night.

Why I was in Hong Kong

I did research at the Hong Kong University of Science and Technology (HKUST) with Professor Gary Chan, whose group is building an AIoT-enabled real-time traffic management system for mega events at the newly developed Kai Tak Sports Park. To optimize traffic around a venue like that, a traffic optimization engine needs several live data sources: car counts, intersection behavior, and people counts. My project focused on people counts, a computer vision task called crowd counting. The standard approach has a neural network produce a density map of an image, and the sum of that map is the number of people in the scene.

The cameras at Kai Tak are a mix of fixed cameras and per-event cameras, so the counting model has to work on views it was never trained on. This is called domain shift. A model that scores well on a proven benchmark can still struggle when it is pointed at a new street with different lighting, angles, or weather.

The plan

Two prior papers from HKUST each address part of this. MPCount (Peng and Chan, CVPR 2024) generalizes to unseen scenes for still images, and Flow-Count (Sizhe Song's MPhil thesis, 2023) uses the motion of people as an extra training signal for video. Neither covers the other's problem, so my goal was to combine them into a video model that uses motion and generalizes to new cameras. I planned to measure it on FDST, a standard video crowd counting benchmark.

What I built

I started by replicating MPCount, then extended it to take a window of three video frames. The new temporal encoder sits before the memory bank, so the part of the model responsible for domain invariance works with motion-aware features instead of static appearance. At the start of training the video model behaves exactly like the original single-frame one, which means any improvement afterward can be credited directly to what I added.

I compared four ways of combining frames: the center frame only (the control), plain averaging, a learned convolution over time, and a convolution over space and time. The learned temporal convolution won, cutting test error from 1.7951 to 1.3712 MAE, about 24% lower. (MAE, or mean absolute error, is the average number of people the model miscounts per frame, so lower is better.) It also beat averaging (1.6627), so the gain came from learned temporal structure and not from smoothing.

Flow supervision took more work. FDST has no motion labels, so I generated the targets myself, estimating optical flow and sampling it at each annotated head. Drawing one arrow per head on the video frames caught a sampling problem that no training curve would have shown me: some people who were clearly walking were being labeled as stationary. After the fix, confirmed misses dropped from twelve to one. In a controlled comparison against an identical model with the flow weight set to zero, the control did slightly better on validation (1.311 vs 1.359 MAE), while the supervised model won on the benchmark test set.

Testing on new cameras

Next I built a cross-domain test from the Mall and UCSD datasets, matched to FDST in crowd density first so that a difference in crowd size wouldn't be mistaken for domain shift. Error on these unseen cameras was higher than on FDST, which pointed me to two things worth understanding.

The first was scale. Doubling the resolution of FDST's own test footage, with no change of camera, raised error from 1.7951 to 11.7980, so the model had learned what a person looks like at one particular size. Correcting for scale showed that about 30% of the cross-domain gap came from scale mismatch and the other 70% from genuine domain shift. Sweeping the input scale on Mall shows error is lowest at 1.5×, which is how I estimated the scale share of the gap.

The second was the benchmark itself. FDST's 100 videos come from about 20 fixed camera locations, and the official split divides clips, not cameras, so all 20 locations appear in both training and testing. FDST was built for video crowd counting, not for testing new cameras, so a good score there tells you how a model does on familiar views. This taught me to focus more on the training domain, and to validate whether the data I'm training on is sufficient for the goal.

Data Acquisition

Since I was in Hong Kong, I was fortunately able to visit the Kai Tak Sports Park and see where this traffic system would be implemented. I managed the cameras recording the event during entrance and exit times. The events were definitely “mega” and the crowds were ginormous. This footage will be used to further improve and validate the model I made.

Did it generalize?

The cross-domain results struggled to transfer. I went back to the original evaluation logs to check, and these are the MAEs at each dataset's native resolution:

Every change I added did slightly worse than the plain single-frame baseline on the new cameras. The temporal convolution, which cut error on FDST by about 24%, did the worst on Mall. Part of what looked like a better model was a model that had learned FDST's familiar camera views more closely.

That taught me a great lesson about overfitting when training machine learning models. Despite the changes I added, the model was still trained on a narrow source domain, and the width of that source domain mattered more than the architecture. I realized a model can only be as general as the range of scenes it learns from.

What I learned

I came into the summer unsure which part of the broader traffic management project I would work on. I met with Aaron and Jason, two exceptionally helpful project managers, and they helped me decide on model training. From there I had a very high degree of independence, choosing what to work on and how to approach it. I learned a lot about training AI models, especially optimizing training architectures through code.

The lessons that will last are about checking things. Before I trust a number, I now check what the test holds out. Before I trust my own pipeline, I look at the pictures and not only the loss. And before I trust an improvement, I test it on cameras the model has never seen, because a model that looks better on familiar data can be worse everywhere else. Building a strong model starts with what it learns from, not just how it is built.

HKUST was a wonderful place to do research, and the UG Summer Research Program wrapped up with a final research symposium where I presented this work.

What's next

The cross-domain results changed what I think comes next. Adding more to the architecture did not make the model more general, so the priority now is widening what it learns from: training across more scenes and camera views, including the footage from Kai Tak, and judging every change on cameras the model has never seen before trusting its FDST score.

The next step is motion and frame-rate augmentation, so the model learns what stays the same across motion patterns instead of details specific to FDST. I also want scale variation in training, and evaluation on held-out cameras. After improving the model's accuracy, I will experiment with making the model work on more resource constrained devices.