For part one, please visit: engineering.footballradar.com/from-a-reac…. Note that the results of this first post have been invalidated by some unfortunate mistakes, but this should set the context for this post.
Monday 23rd May we posted the results of a performance experiment using React and Vue. The idea was to work out whether React or Vue would be more suitable for tabular data that updates a lot, especially when performance is critical. This is very representative of some challenges we work on at Football Radar.
Initially we were very encouraged and surprised by the results, but a few important mistakes meant that our "findings" were not as revelatory as as we thought. Thanks to some great feedback from both the React and Vue communities - especially React core engineer Christopher Chedeau (@vjeux) and Vue creator Evan You (@youyuxi) - we were quickly able to narrow down the problems with our tests. We're very grateful for the high-quality discussion that followed; I'm slightly embarrassed that these mistakes made it into the open but we've learnt a lot from the responses, so it's been a blessing in disguise.
Considering how many improvements have been made as a result of the first post there is nothing to say there can be no further improvements. With that in mind this post is more of a document of a journey than a conclusive, end result.
First of all I would like to recap the intention of this experiment, then talk about the mistakes we made - giving credit to those who helped us fix them - and finally post a fairer set of results from the updated tests.
The Test
The experiment was to output a list of football games, each updating its data once per second. In order to test performance and scaling characteristics, we altered two independent variables: number of games and delay between updates.
In order to test the performance, we tested with 50, 100 and 500 games in both Vue and React, with delays of 100ms and 1 second. The idea was to simulate the conditions of load on the page and see how both implementations scale.
Our first test suggested that Vue massively outperformed React, which was very sensational and unexpected, but was caused by running the tests in development mode. This was the biggest omission and definitely the main cause of the skewed results. Thank you to Christopher Chedeau (@Vjeux) for catching this bug: github.com/footballrad…. By running React in production mode, many expensive parts of React are avoided, including prop-types checking and warnings. This is an obvious optimisation, but since React runs in development mode by default, one that is easily overlooked. It should be stressed that Vue was also running in development mode, so both tests were equally afflicted. The discussion that followed in that pull request was particularly insightful.
We also pushed some broken test data that further skewed the results by having Vue perform 10x as much work as the equivalent version in React. In our original tests both frameworks were run against equivalent data, but the version on Github had this inconsistency, so other people running the tests may have seen misleading results.
Changes from previous test
New Findings
We'll post the timeline summaries below and we have screenshots of the actual timeline and bottom-up stacks in the repo: github.com/footballrad…
Interestingly the Chrome dev tools crashes when retrieving a 30s timeline for the Vue 500 game tests but doesn't for React 500 game tests. We captured 15 seconds instead and that was even more confusing as Vue has a lot more idle time than React.
All of the results below share a common theme: React performs more scripting work, as we would expect from its virtual DOM implementation, and Vue is heavier on painting and rendering as it touches the DOM directly. The key difference is that in terms of total work done, Vue still appears to be faster by up to 25% in some cases. This is not the huge difference we originally reported but still noteworthy.
React:
Vue:
React:
Vue:
React:
Vue:
React:
Vue:
React:
Vue:
React:
Vue:
Conclusion
Overall, the original claims about Vue's performance still hold some value in this use case but there was clearly a lot of opportunity for optimisation - especially with React. One surprising take away is how much work and collective knowledge was required to bring out better performance from React, while Vue was fairly well optimised from the get-go. Whatever we can say about the relative performance of these two libraries, this is definitely a win for Vue's developer experience.
