The COVID-19 pandemic changed the life of many. The study aimed to identify fragile groups of the population regarding nutrition during the COVID-19 restriction period. An online survey was conducted from 26 July to 3 September 2020. A total of 1173 answers were received from persons aged 18–68 years living in Latvia. The results showed that food availability mainly was not limited, although for 15.3% responders restrictions caused increased worries about not having enough food, and 12.2% noted a decrease in the ability to eat preferred food. Responders aged 40–54 noted that they ate less, and that their food stocks were scarce in comparison with other age groups. Almost 50% of responders increased food consumption. One-fifth of responders increased consumption of foods of low nutritional value, especially among 25–39 year old persons and in households where the economic situation became worse. Fragile groups regarding nutrition in times of COVID-19 restrictions included households whose economic situation became worse during the COVID-19 restrictions, younger people who increased food consumption together with consumption of food of low nutritional value, and persons aged 40–54 years. The data from the study serve as an indicator that more detailed research is needed to determine whether crises such as the COVID-19 pandemic are changing dietary habits and food availability in the population.