Every day, our species chews its way through more than a million terajoules of energy. That’s roughly equivalent to what we would use if all 7.5 billion of us boiled 70 kettles of water an hour around the clock. Or 3,000 times the daily output of Palo Verde nuclear power station in Arizona – one of the world’s largest – running at full capacity.

With the global population swelling and industrialisation on the rise in developing nations, humanity’s hunger for energy has reached unprecedented levels. More than half of our energy comes from fossil fuels extracted from deep within the Earth’s crust. It is estimated that since commercial oil drilling began in the 1850s, we have sucked up more than 135 billion tonnes of crude oil to drive our cars, fuel our power stations and heat our homes. That figure increases every day.

But our gas guzzling over the past two centuries has taken a potentially devasting toll on the planet. Burning of coal, oil and gas has been inextricably linked to the rising levels of greenhouse gases in Earth’s atmosphere and is a leading contributor of climate change. The world’s scientists agree that we are on a path towards disaster that can only be stopped by weaning ourselves off our fossil fuel habit. But that leaves us with a problem. How do we ensure the lights stay on?

“The energy industry is facing decades of transformation,” according to a recent report by the World Energy Council. Yet the implications of the changes underway go far deeper. There are political, economic and social issues at stake, but it may also require each of us to make some fundamental shifts in our behaviour too.

There can be no doubt that implementing a shift in where we get our energy from is one of the grand challenges facing our planet today. BBC Future Now spoke to a panel of experts about what hurdles we must now overcome and where technology may provide an answer.

Perhaps the greatest issue raised by the scientists, policy experts and companies we spoke to is how to cope with the immediate hike in energy demands expected in the coming decades.

“There are still a lot of people around the world – 1.2 billion or so – who do not have access to modern energy services,” explains Jim Watson, director of the UK Energy Research Centre. “There is going to be a lot of rising demand from regions like Asia, Latin America and parts of Africa.”

‘There is going to be a lot of rising demand from regions like Asia, Latin America and parts of Africa’ – Jim Watson, director of the UK Energy Research Centre

There are still an estimated three billion people around the world who cook and heat their homes using simple stoves or open fires that burn wood, animal dung or coal. As developing nations become more industrialised, they will need access to reliable electricity supplies. In countries where development is already underway, energy use will soar as increasing wealth leads to a swelling middle class and the lifestyle trappings that brings with it.

“Globally, the greatest challenge for energy is going to be cooling,” says Martin Freer, director of the Birmingham Energy Institute at the University of Birmingham. “With the growth of the middle class in India and China, there will be an associated demand for air conditioning. The United Nations’ Intergovernmental Panel on Climate Change suggest that by the middle of the present century, the demand for cooling will outstrip the demand for heating.” Indeed, it is estimated that by 2040, the world’s energy consumption will have increased by almost 50%.

‘Globally, the greatest challenge for energy is going to be cooling’ – Martin Freer, director of the Birmngham Energy Institute, University of Birmingham

But faced with global agreements to reduce the amount of carbon dioxide being released into the atmosphere, how will we meet this growing demand without dooming our ice caps and drowning low-lying regions beneath rising sea levels?

In truth, the picture may not be as bleak as it could be. Around a fifth of the world’s primary energy supply already comes from renewable sources such as wind, solar, hydro and geothermal. This sector is expected to continue growing by 2.6% each year until 2040.

Until recently, the main source of renewable energy was hydroelectric power and wind has been the fastest growing. But new advances in solar panel technology, which allow them to generate power even in overcast conditions, have seen a surge in the amount of energy being produced using the sun.

In the UK, for example, more than 12GW of solar energy has been added to the electricity network in the past 12 months – the equivalent of an entire coal-fired power station. Worldwide, the amount of solar energy production grew by 50% last year. Researchers in many countries are working on new photovoltaic cells that can be printed on flexible sheets, which could reduce the costs of solar further.

Hooking up these new energy producers to existing grids won’t be straightforward, however. “One of the big challenges of deploying these intermittent renewables like wind and solar is the impact they could have on the system,” says Watson. In many Western countries, the networks that carry our electricity supply into our homes and offices are decades old, designed to deal with steady, reliable power generation. Wind and solar energy are highly dependent on the weather – and the time of day, in the case of solar – meaning they do not necessarily produce the bulk of their electricity at times when there are peaks in demand.

“It used to be that the summer was a really quiet time for the grid operator compared to the winter,” explains Watson. “Now they are having this peak in generation in summer due to solar energy when demand is low. They are having to juggle this as we cannot store electricity in large quantities yet. This is a new way of operating for them.”

Most countries are currently tackling this by keeping more reliable sources of energy production in reserve. This means having nuclear, gas and even coal fired power stations sitting idle or running at a low level, but ready to ramp up their production should the wind drop or when the sun dips beneath the horizon.

According to Robert Armstrong, director of the Massachusetts Institute of Technology Energy Initiative, this puts limits on how much renewable energy you can practically use. Armstrong’s models suggest that without energy storage only about 10% of our power could come from solar. “The reason is that solar is concentrated around midday, so you need generation to match demand in the evenings and the mornings. There are issues around who builds that and who pays for that.”

‘Transferring electricity from regions that need it least to those that need it most would help to enable substantial economic benefits’ – Ksenia Letova, Skoltech Institute for Science and Technology in Russia

One solution to this is to make the grids that distribute the electricity bigger – create so-called “supergrids”. The basic idea is that if energy is shared over a wider area, there is more chance that the sun will be shining or the wind will be blowing in one part of a supply network, if not another.

These schemes envisage connecting the energy grids of several countries together so electricity can be shared between nations. Proposals for a European supergrid and one in the United States have been discussed for decades. More recently there have even been calls for a global energy grid – an idea that has had support from Chinese State Grid, which set up the Global Energy Interconnection Development and Cooperation Organization.

There are already some moves in this direction. The UK is building new underwater connections to energy grids in France, Belgium, Denmark, Ireland and Norway, with a capacity to import or export up to 11GW of electricity. There are also moves towards building an Asian supergrid which will connect Japan, Russia, China, Mongolia and South Korea.

“Transferring electricity from regions that need it least to those that need it most would help to enable substantial economic benefits,” says Ksenia Letova, project manager of the Asian Supergrid at Skoltech Institute for Science and Technology in Russia. “In countries like Japan and South Korea, the maximum seasonal load falls on the summer because of extensive use of air conditioning. In the Russian Far East and Siberia this is the period of the lowest electricity demand.”

Using the excessive capacities of neighbouring countries may help to reduce costs of building new energy projects. For example, there are plans to develop large-scale wind and solar power stations in the Mongolian Gobi desert and in northern regions of China. These regions are sparsely populated but allowing the excess energy produced to be exported could bring in significant revenue.

“The problem is the cost of building such a grid,” says Janusz Bialek, director of the Centre for Energy Systems at Skoltech Institute of Science and Technology. Transmitting electricity over long distances can be inefficient and many countries will need to upgrade their powerlines to cope.

‘Any failure of, or attack on, a global grid could have very serious consequences, threatening supply in many countries’ – Janusz Bialek, Skoltech Institute of Science and Technology

“There are political considerations too,” says Bialek. “Any failure, or attack on, a global grid could have very serious consequences threatening security of supply in many countries.”

It is an important point. Energy security is already the main driving force in the geopolitical landscape. Countries with large oil reserves can largely dictate global policy and nations like Saudi Arabia have won themselves powerful allies like the United States as a result of the black gold buried beneath their territory.

Russia has also used its fossil fuel resources to flex its muscles in recent years, turning off gas supplies through Ukraine, and so to several European countries, in disputes over prices and debts. Nearly a quarter of the natural gas consumed in the European Union comes from Russia.

Yet as nations become more reliant upon alternative sources of energy, it could also see these traditional power struggles changing. The largest producers of oil in the world are Saudi Arabia, the US and Russia. By contrast, currently the largest producers of solar energy are China, Germany and Japan while the US, China and Germany are the world leaders in wind energy.

“As new technology is developed it will shift the geopolitics of energy,” says Watson. “It will change relationships.”

‘As new technology is developed it will shift the geopolitics of energy, it will change relationships’ – Jim Watson, director of the UK Energy Research Centre

With countries such as Morocco building giant solar farm projects in their vast areas of desert in the hope of exporting this vast resource to other countries, it could lead to states that have previously been small fish taking a bigger role on the global stage. If Mongolia builds its huge reserves of wind and solar in the Gobi, it could transform its standing in the eyes of the world.

Yet the need for supergrids and power-sharing deals between countries would be diminished if a good way to store electricity can be found. Beyond their widespread availability and affordability, fossil fuels have one major advantage over renewable energy sources – they are very easy to store and transport.

Currently there is no easy way to store the electricity produced by wind or solar energy for appreciable periods of time. Technologies like capacitors and flywheels can provide stored energy for a few minutes or hours. But electricity grids need to be finely tuned. They only work when the amount of energy put in is the same as that drawn out. The supply must match the demand.

Many are still scratching their heads about what to do when the sun is shining and the wind is blowing yet electricity demand is low. Hydroelectric dams can provide some of the solution. In the UK, excess electricity is used to pump water to the top of these dams in Scotland and Wales, where it is stored as potential energy. When it is needed, the water is released to drive turbines.

Cables currently being laid under the North Sea will also soon allow the UK to access the large amounts of hydroelectric storage in Norway. Excess energy from wind and solar will be exported to Norway to be stored before being bought back off them when needed.

But building new hydroelectric dams is controversial and extremely damaging to local habitats. It has left researchers searching for another solution. Some are looking at building banks of batteries to store this energy, but battery technology is not yet good enough to efficiently store large amounts of energy.

“My guess is that the solution is going to come in the form of fuel,” says Armstrong. “We can make fuel out of excess solar or wind, perhaps by splitting water to produce hydrogen and possibly taking some of that excess energy and reducing carbon dioxide to combine with the hydrogen to make synthetic hydrocarbons.”

‘The solution to storing energy is going to come in the form of fuel’ – Robert Armstrong, director of the Massachusetts Institute of Technology Energy Initiative

Currently there are a number of small scale projects around the world attempting just this. Aberdeen in Scotland, for example, is running the world’s largest demonstration project of hydrogen fuel cells in buses. Renewable energy is used to generate hydrogen, which is used to power 10 public buses around the city.

This could also solve another of the pressing problems facing many countries with colder climates – how to stay warm. “One of the big challenges that countries like the UK has is how do you make heating more sustainable,” adds Watson. “These countries are still using fossil fuels for heating. You might need storage that lasts several months. This is an area that is really ripe for innovation and we are really only at the start of deploying and testing potential solutions.”

Most countries in northern Europe, including the UK, use natural gas to heat their homes. If Britain was to abandon gas and use electricity for heating, it would require a four- or five-fold increase in the capacity of the electricity network overnight, according to Julian Leslie, head of electricity network development at the UK’s ‎National Grid.

‘Increasing the capacity of the electricity network is going to be hugely expensive’ – Julian Leslie, head of electricity network development at the UK’s ‎National Grid

“That is going to be hugely expensive,” he says. “You would struggle to get the planning consent for the wires and generation needed. There is a very strong future for gas and we can decarbonise the gas a lot by producing hydrogen or biogas to inject into the gas network. We need to explore more alternatives to how we can decarbonise gas further.”

Biogas and biofuels are often seen as one of the most viable alternatives to fossil fuels, with companies like BP investing huge sums into developing production lines. Yet burning these fuels will not halt the release of greenhouse gases into the atmosphere. And with the burgeoning extraction of shale oil and gas in many parts of the world, it seems likely we will be reliant on fossil fuels for some time to come.

“By 2050 we will still be getting 75% of our energy from fossil fuels,” says Armstrong. “A critical issue for us will be to figure out how to reduce carbon dioxide emissions from those energy sources. That is going to require carbon capture storage and utilisation. Lowering the cost of capturing the carbon is probably the toughest piece of that but we also need to figure out how to store it for geological timeframes.”

‘By 2050 we will still be getting 75% of our energy from fossil fuels’ – Robert Armstrong, director of the Massachusetts Institute of Technology Energy Initiative

Carbon capture involves scrubbing the carbon dioxide gas from the fumes leaving chimneys. Typically this involves expensive chemicals to bind the carbon dioxide and requires replumbing power plants so the mixture can be heated as part of the scrubbing process. But there are some new approaches being developed that make use of metal ions, which avoids the need to heat the chemical mixture.

Some believe the problems we face with renewable energy can be overcome in more direct ways, such as closer monitoring of individual households’ energy consumption. By 2020 the European Union is aiming to have 500 million smart meters installed in homes to monitor energy usage.

Detailed minute-by-minute information about demand should help power companies manage grids better. “Artificial intelligence will be essential in analysing the vast amounts of data generated around the power grid and taking real-time control decisions,” says Valentin Robu, a lecturer in smart grids at Heriot-Watt University in Edinburgh, Scotland.

It could also lead to some fundamental changes in the way we consume technology. “We are not that far away from me asking my energy company for the cheapest possible tariff they can give and them sending me some plugs that connect to my WiFi network,” says Leslie. “It will mean that you are no longer in control of your dishwasher or your washing machine.”

‘It will mean that you are no longer in control of your dishwasher or your washing machine’ – Julian Leslie, head of electricity network development at the UK’s ‎National Grid

Instead, it will mean our energy companies will control when our household appliances run. They will be able to turn them on when the weather is sunny and solar energy is plentiful or when demand is otherwise low.

Such approaches would mean a fundamental move away from the “power-on-demand” way of consuming energy we have grown use to over the past century. Instead when our fridges go through cooling cycles or appliances turn on could be determined by fluctuations in the weather or the time of day. Our dishwashers, for example, may run during the day while we are at work rather than at night.

“For years we have consumed energy whenever we like and paid a flat rate for that,” says Leslie. “We will have to start shifting our use of energy to when it is there and available rather than shifting the energy production to match our use.”