Analysis and prediction of second-hand house price based on random forest

Web crawler, also known as web spider [8], is a program that can automatically browse web pages, grab web page information, analyse and store it. The key technologies of web crawler are mainly reflected in three aspects: Web page acquisition, web page analysis and data storage [9].

Today is the era of big data, and data analysis is applied to all aspects of society. Data analysis is generally embodied in three aspects [10]. (1) Data pre-processing. The raw data obtained may have irregular problems, such as missing data or data duplication, and thus data pre-processing is required before data analysis. (2) Descriptive analysis. Some laws of the data can be displayed intuitively through graphs, tables, etc. Matplotlib and Pandas will be used in the analysis performed in the current study. Matplotlib can generate data visualisation modules to help users draw. Pandas data analysis package can quickly provide structured data resources to Python. (3) Predictive analysis. Some laws discovered through descriptive analysis are processed by feature engineering, and mathematical statistical algorithms, machine learning models, deep learning models, etc. are selected to predict the development of things.

The data come from Beike Network (https://www.cd.ke.com). Beike Network is an online estate trading platform created for new houses, rented houses and second-hand houses. The housing data that need to be collected mainly include basic information (house type, floor, building area, housing orientation, etc.), transaction attributes (transaction ownership, housing use, housing age), housing characteristics, etc.

There are only 30 listings on each page. If you want to get more listings, you need to jump to the next page. Observing the composition of the url, we can find that the pg of each page increases by 1, and thus the url can be spliced into ‘https://www.cd.ke.com/pg’+str(page)+’/’, controlled by the ‘for loop’. The 30 detailed urls on each page are located through beautiful.find_all(‘li’) and then saved in the list. Through the url in the list, you can enter the source information details page of each house, locate the required field information and then save it in the CSV file. The specific process of the crawler is shown in Figure 1.

Fig. 1

When crawling, the collected data are crawled by region, one table per region. Therefore, you first need to combine the data of all regions into one table using pandas.concat(). After the data are merged, the basic information column of the house contains multiple pieces of information, such as the house type, floor, building area, building structure, decoration and orientation, etc., and these are extracted using the re.findall() function to form a new column, which is the community profile column. The building age is extracted using the same method and becomes a new column; the regular information collected matches the basic information column of the house and the characteristic column of the house, including whether there are elevators and subways, and the results are assessed in terms of whether there is positive affirmation, which is given by 1, and no is 0. Finally, we delete the unnecessary columns such as community name, house number, basic house information, house transaction information, community brief introduction and house characteristics. The reprocessed data are shown in Table 1.

We use the pandas describe() function to perform descriptive analysis on the data set, and the analysis result is shown in Table 2. It can be seen from the table that the average total price of second-hand houses in Chengdu is 1.49 million yuan, and the average area is 94 m². Dividing the former by the latter, the average price is obtained as about 15,900 yuan/m², which is not much different from the average unit price in the table. It shows that the abnormal value of the data is not obvious, and the next step of data visualisation analysis can be carried out.

Housing prices are used as dependent variables. Also, area, bedroom, hall, floor, elevator, subway, orientation, building structure, decoration and construction age are used as independent variables. We study the influence of independent variables on dependent variables.

4.2.1

Regional Impact on House Prices

The average price of houses in each area is shown in Figure 2. As can be seen from the figure, the top three areas with average unit price are High-tech Zone, Tianfu New Zone and Jinjiang District. Tianfu New District has great potential for development; it is a national-level new district with the theme of a park city, which makes people come in an endless stream, and thus the second-hand housing market is also developing. The high-tech zone is a bridge connecting the old city and Tianfu New District, new houses are in short supply and the secondhand housing market is also rising. There are many scenic spots near Jinjiang District, and life and entertainment are very convenient. Not only is Xinjin District far from the main urban area but also the current development there is not prominent among the districts, and thus the housing prices are the lowest.

Fig. 2

4.2.2

The Impact of the Number of Bedrooms on House Prices

The price per unit area for various numbers of bedrooms is shown in Figure 3. From the figure, we can see that the more bedrooms there are, the higher the house price. When the number of bedrooms is six, the price is the highest. When the number of bedrooms is seven, the impact on the house price is reduced. The number of bedrooms in most houses is below six; the ones that exceed this number are generally villas or old houses, and the house prices of old houses will be lower. In the data collected during this time, most of the seven bedrooms are old houses, and thus it becomes clear as to why the number of bedrooms has increased despite the fact that the house price has decreased.

Fig. 3

4.2.3

The Impact of the Number of Halls on House Prices

The price per unit area of various numbers of halls is shown in Figure 4. One and two halls have the highest price per unit area; the lowest price per unit applies in the case of five halls. The rest have less impact on housing prices. The vast majority of second-hand houses on the market are those with one hall or two halls. If there are more rooms, the room type will be unacceptable.

Fig. 4

4.2.4

The Impact of Floors on House Prices

Various numbers of floors have a certain impact on house prices, but the impact is not particularly significant. Generally, people think that the house price on the middle floor is the most expensive, but on this occasion the data show that the house price on the lower floor is the highest. The price per unit area of various floors is shown in Figure 5.

Fig. 5

4.2.5

The Impact of Orientation on House Prices

Figure 6 shows the number of houses with various orientations. It can be seen from the figure that the number of houses facing south and southeast is the largest: they constitute more than half of the market. China is in the northern hemisphere, a house facing south is conducive to receiving more sunlight and the sunshine time is longer.

Fig. 6

The housing prices per unit area of various orientations are shown in Figure 7. The housing prices for houses facing north and west are the most expensive, and the difference in housing prices between houses facing other directions is not obvious.

Fig. 7

4.2.6

The Impact of Decoration on House Prices

The decoration of second-hand houses can be categorised mainly into simple decoration, hard decoration and rough decoration. Hardcover rooms have the greatest impact on housing prices, and there is a certain difference between rough and simple rooms. The price per unit area of various decoration conditions is shown in Figure 8.

Fig. 8

4.2.7

The Impact of Building Structure on House Prices

The housing prices per unit area of various building structures are shown in Figure 9. The steel-concrete and frame structure houses are the most expensive. Reinforced concrete has good seismic performance and durability. At the same time, the cost is higher. Many high-rise and century-old buildings use steel-concrete structure. Frame structure houses are also very good in terms of seismic and impact resistance, and the cost is a bit cheaper. Therefore, the housing structures on the market are largely composed of steel-concrete and frame. The rest of the building structure has a small market share and causes a small impact on housing prices.

Fig. 9

4.2.8

The Impact of Elevators and Subways on House Prices

The question of whether there are subways and elevators has an impact on the price per unit area, as shown in Figures 10 and 11. As can be seen from the figure, elevators and subways have a great impact on housing prices. Elevators and subways provide great convenience to people's daily lives and transportation. In today's society, people will take these two factors into consideration when buying a house, and the same is true for the second-hand housing market.

Fig. 10

Fig. 11

4.2.9

The Impact of Construction Age on House Prices

The influence of construction age on the price per unit area is shown in Figure 12. From the figure, we can see that since 2000, housing prices have basically shown a linear upward trend. But 2016–2018 house prices are on a downward trend. Investigating the reasons, it was discovered that during this period, there was a steady supply of new housing, and the price-limited housing under policy control was also cheaper. The demand for second-hand housing was small, and the price also fell. Since 2018, housing prices have continued to rise, and more and more people are buying houses by lottery. Those who are anxious to buy a house but don’t have enough deposits turn to second-hand houses. As a result, market demand for second-hand housing increased, and housing prices continued to rise.

Fig. 12

4.2.10

The Impact of Area on House Prices

The relationship between the area and the total price of second-hand houses is shown in Figure 13. House area and total price are positively correlated. The larger the area, the higher the total price. In this area, with a housing area of 0–300 m² and a total price of 0–10 million yuan, the distribution of data points is relatively concentrated.

Fig. 13

RF uses the idea of integrated algorithm to randomly form a forest of multiple decision trees, each decision tree is independent of every other and the trees have no correlation [11, 12]. These decision trees are all built for the same task, and their results are finally averaged.

The key to RF is randomness [13, 14]. If the model of each tree is the same, then the average result is also the same. We need to make the model diversified. The first is the randomness of the data set, and sampling with replacement is adopted. For example, the first decision tree randomly selects 80% of the data as input data, and the second decision tree also randomly selects 80% of the data as input data, so that the input data of each tree is different. The second is the randomisation of features, and each tree randomly extracts 60% of the features. The flow of the RF model is shown in Figure 14.

Fig. 14

This paper uses the function train_test_split() to divide the data set into a training set and a test set at a ratio of 3:1. The training set data are used to train the RF model to predict the unit housing price in the test set. The evaluation standard of this model is the mean absolute percentage error (MAPE), and the calculation formula is as follows: (1) MAPE=∑i=1N| yi−predictediyi |×100%N MAPE = \sum\limits_{i = 1}^N {\left| {{{{y_i} - {predicted_i}} \over {{y_i}}}} \right| \times {{100\% } \over N}} where y_i represents the actual unit price and predicted_i represents the predicted unit price. The smaller the value of MAPE, the better the prediction effect of the model.

Feature engineering is the process of transforming the original data into model training data, and this process involves extracting features from the original data to the greatest extent. Applying these features to the prediction model can improve the prediction accuracy of the model [14]. The better the feature selection and preparation, the better the results obtained.

5.2.2

Feature Selection

After feature processing, feature selection is required [15]. These 11 characteristic variables are manually selected based on experience. However, these characteristic variables may not necessarily improve the accuracy of model prediction. In fact, some variables have little effect on model prediction, but it takes more time for the effect to become prominent. This paper selects feature variables by calculating feature importance.

Feature importance is to find the contribution of each feature on each decision tree in the RF, then take the average and finally compare the contribution between the features. This contribution uses the Gini index [16] as the evaluation index, and the calculation formula is as follows: (2) GIm=1−∑k=1| k |pmk2 {GI_m} = 1 - \sum\limits_{k = 1}^{\left| k \right|} {{p_{mk}}^2} where K indicates that there are K categories and P_mk indicates the proportion of category k in node m.

The exponential change of feature x j before and after branching on node m is (3) VIMjm=GIm−GIA−GIB {VIM_{jm}} = {GI_m} - {GI_A} - {GI_B} where GI_A and GI_B represent the Gini index of the two nodes after the branch of node m.

The node of feature x_j in decision tree i is set as M, and there are N decision trees in the RF; then, the importance of x_j is (4) VIMj=1N∑i=1N∑mεMVIMjm {VIM_j} = {1 \over N}\sum\limits_{i = 1}^N {\sum\limits_{m\varepsilon M}^{} {{VIM_{jm}}} }

We use Python's feature_importances_ function to find the importance of each feature variable, as shown in Figure 15.

Fig. 15

5.2.3

Feature Combination

Based on the calculation results of the above feature importance, three feature selection schemes are set up next: (1)

Select all characteristic variables;

(2)

Select the top six feature variables of importance;

(3)

Select feature variables whose feature importance exceeds 94% after accumulation.

We use the cumsum() function to calculate the accumulated value. It can be seen from Figure 16 that after the feature importance is accumulated, more than 94% of the feature variables are district, elevator, area, building age, bedroom, direction and building structure.

Fig. 16

The modelling time of the three schemes applied on the training set and the effect of the predicted error values on the test set are shown in Table 4. It can be seen from Table 4 that the modelling time of the third solution proposed in this paper is greatly shortened, and the error value is reduced. Therefore, we can conclude that the most critical factors affecting housing prices are district, elevator, area, building age, bedroom, direction and building structure.

Table 4

Comparison of three schemes

Adopted scheme	Time (s)	Predicted error values (%)
Scheme1	37.37	4.83
Scheme2	37.24	4.84
Scheme3	21.91	2.62

In the RF model, there are a total of 17 parameters. Optimising the parameters can improve the accuracy of the model. Here, we select several parameters that have a great impact on the model, such as n_estimators (number of trees) and max_ Features, Max_ Depth, in_ samples_ Split (the minimum number of samples required for node splitting) and min_ samples_ Leaf (the minimum sample number of leaf nodes) for tuning. Due to time and performance reasons, we limit the value range of the given parameters in this paper. The value range is shown in Table 5.

The experiment uses the grid search algorithm (GridSearchCV) in the sklearn library to tune the parameters. For the grid search algorithm, each possible parameter combination is used as a parameter input for modelling, the optimal parameter result is given and then the best parameter combination is determined through cross-validation. The optimal parameter combination obtained by the grid search algorithm is shown in Table 6.

The characteristic variable combination of Scheme 3 in Section 5.2 is used as the input of the RF model, and the optimal parameter combination and the default parameter combination are, respectively, used for comparative experiments. The error results of prediction on the test set are shown in Table 7.

It can be seen from the results that the optimal parameter combination further improves the prediction effect of the model.

The adjusted RF model is used to predict the second-hand housing prices in Chengdu, and the prediction results are obtained. For example, a user wants to buy a second-hand house in 2015, with an area of 95 m², having elevators and subways and located in Jinjiang District, with three bedrooms and two halls, west-facing, having a steel-concrete structure and mid-floor, and being a hard-decorated second-hand house. The model predicts that the unit price is 27,065.8 yuan/m², and the total price is 2,571,300 yuan. The user has rigid requirements for the area, house type, and elevator and subway, but the budget is not enough. Based on the predictive analysis, the building structure, orientation and building age have a greater impact on the housing price. According to the visual analysis, corresponding to the attributes of frame structure, south facing and constructed in 2018, a lower annual average unit price is established, so that users can adjust according to their own situation and choose the most cost-effective second-hand housing.