How to write byte array without OOM?
I have an android app which handles some large byte array but I am getting some OOM crash in my Firebase Crashlytics reports for devices with low memory while handling byte array whose size may go from 10 mb to 50mb. Below is my method that I have used. So could anyone help me to improve it to avoid OOM.
byte[] decrypt(File files) < try < FileInputStream fis = new FileInputStream(files); SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(), "AES"); Cipher cipher = Cipher.getInstance("AES"); cipher.init(Cipher.DECRYPT_MODE, sks); CipherInputStream cis = new CipherInputStream(fis, cipher); ByteArrayOutputStream buffer = new ByteArrayOutputStream(); int b; byte[] d = new byte[1024]; while ((b = cis.read(d)) != -1) < buffer.write(d, 0, b); //this is one of the line which is being referred for the OOM in firebase >byte[] decryptedData = buffer.toByteArray();//this is the line which is being referred for the OOM in firebase buffer.flush(); fis.close(); cis.close(); return decryptedData; > catch (Exception e) < e.printStackTrace(); return null; >> EDIT Actually I am using the above method for decrypting downloaded audio files which are encrypted during downloading. The above methods return the content of the encrypted files to exoplayer to play its content and I am calling the above method in the following way
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file)); Uri uri = new UriByteDataHelper().getUri(decrypt(some_file)); DataSpec dataSpec = new DataSpec(uri); src.open(dataSpec); DataSource.Factory factory = new DataSource.Factory() < @Override public DataSource createDataSource() < return src; >>; audioSource = new ProgressiveMediaSource.Factory(factory).createMediaSource(uri); 3 Answers 3
First of all, I would make sure that the devices where this is running have enought heap memory to run this, it might be that simply the software has already been allocated with a lot of space and there might not be much more left on the heap to provide the software. This operation should not require much memory and I don’t see anything obvious that would point towards trying to allocated and unexpectadely large amount of memory.
What I would recommend though, if you want to hav a quick test is actually simply lowering the byte array size, any particular reason why you are using 1024? If possible perhaps try:
Also, If that was me, I would store the read data temporarily, on an array perhaps and only once the read of the cypher has finished, I would call
From my experience, trying to read and write at the same time tens to not be advised and can result in serveral issues, at the very least you should make sure you have the whole cypher and that it is a valid one (if you have some validation requirements) and only then send it.
Again, this should not be the core issue, the device seems that it is lacking enough availalbe memory to be allocated, perhaps too much reserved memory for other processes?
Could you please help me with a complete code of my above method, if it doesn’t cause much trouble for you
I would try and replace byte[] d = new byte[1024]; with byte[] d = new byte[8]; first of all and see if you witness the same problem. Also, make sure you have a way to monitor you memory available and how uch is available to be assigned
@Tiago The OP is using a ByteArrayOutputStream which is storing all of the decrypted bytes in memory. That is likely the major source of their problem. Not only does the object itself hold an array but getting the array from it creates a copy.
@kmrinmoy07 What are you actually trying to do with the byte[] ? Why do you need everything in memory?
@Slaw Actually I am have some downloaded audio files and to make it only playable in the specified app I have encrypted it. So now to make it playable again I am decrypting it using the above line of codes. Could you help if there is any better way
You should consider of writing the decrypted data to a tempfile and then reload the data for usage.
The main reasons for the Out of memory error are the ByteArrayOutputStream AND byte[] decryptedData = buffer.toByteArray(), because both of them hold the complete (decrypted) data and that doubles the memory consumption of your decrypt method.
You could avoid this by decrypting the data to a tempfile in the first step and later load the data from the tempfile. I modified the decrypt method to handle the decrypted output stream and later there is a method for reloading the decrypted data (there is no propper exception handling and for my testing I setup a static encryptPassword-variable . ).
There is just one part left for you — you need to find a good place for the tempfile and I’m no Android specialist.
Just two notes: You are using the unsecure AES ECB mode and the String to byte[]conversion for your password should be changed to
.getBytes(StandardCharsets.UTF_8) on ecryption and decryption side to avoid errors caused by different encodings on different platforms.
public static void decryptNew(File files, File tempfiles) < try (FileInputStream fis = new FileInputStream(files); BufferedInputStream in = new BufferedInputStream(fis); FileOutputStream out = new FileOutputStream(tempfiles); BufferedOutputStream bos = new BufferedOutputStream(out)) < byte[] ibuf = new byte[1024]; int len; Cipher cipher = Cipher.getInstance("AES"); SecretKeySpec sks = new SecretKeySpec(encryptPassword.getBytes(),"AES"); // static password // SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),"AES"); cipher.init(Cipher.DECRYPT_MODE, sks); while ((len = in.read(ibuf)) != -1) < byte[] obuf = cipher.update(ibuf, 0, len); if (obuf != null) bos.write(obuf); >byte[] obuf = cipher.doFinal(); if (obuf != null) bos.write(obuf); > catch (BadPaddingException | IllegalBlockSizeException | InvalidKeyException | IOException | NoSuchAlgorithmException | NoSuchPaddingException e) < e.printStackTrace(); >> public static byte[] loadFile(File filename) throws IOException < byte[] filecontent = new byte[0]; FileInputStream fileInputStream = null; try < fileInputStream = new FileInputStream(filename); // int byteLength = fff.length(); // In android the result of file.length() is long long byteLength = filename.length(); // byte count of the file-content filecontent = new byte[(int) byteLength]; fileInputStream.read(filecontent, 0, (int) byteLength); >catch (IOException e) < e.printStackTrace(); fileInputStream.close(); return filecontent; >fileInputStream.close(); return filecontent; > After loading the tempfile content to the byte array you can delete the file with a one-liner (again no exception handling):
Files.deleteIfExists(tempFile.toPath()); Yes, it can be done by a tempFile but I would like to go in this way only when there is no any other better option. I hope there is some other way too
@kmrinmoy07: There is indeed another solution — decrypt the data in the (encrypted) byte array after the download not in total but in chunks and write the chunk directly back to the byte array. That way the memory consumption is only as much as the download file. See tutorials.jenkov.com/java-cryptography/cipher.html -> Encrypting / Decrypting Part of a Byte Array and I used the CTR-mode for en-/decryption.
Yes, it can be done but I have one more issue. As already told I am using these byte array as a source to Exoplayer , so now if I pass one chunk as a source I would like to know how to handle the audio forwarding/skipping properly in chunks.
Sorry for my bad explanation yesterday night. The workflow could be 1) load encrypted audio file to your byte array, then (!) decrypt the byte array in chunks (for memory consumption reasons on low memory devices) OR in once on high memory devices. The «chunk version» is writing the decrypted chunks back at the same place in the same byte array and in the end the byte array contains the decrypted data that you can present to the exoplayer with all features(forwarding etc). The maximum memory consumption would be 50mb plus maybe 100 bytes BUT you will need AES CTR mode for this (I tested it)
Could you please post a code of it as a demo so that I can implement, anyways thanks a lot for the help.
I’m writing a second answer and do not edit my first answer as it is a total different approach to solve the problem.
As you post a part of your code I can see that you have a byte array with the complete and decrypted content that gets played by the exoplayer:
output: byte[] decrypt(File files) as input for ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file)); So to avoid double and moretime memory consumption when playing with large files (approx. 50mb) my approach it is to download the complete encrypted file and save it in a byte array.
On devices with a good memory equipment you can decrypt the encrypted byte array in one run to another byte array and play the music from this decrypted byte array (step 6 + 8 in my sample program).
Using a low memory device you decrypt the byte array in chunks (in my program in 16 byte long blocks) and save the decrypted chunks at the same place in the encrypted byte array. When all chunks got processed the (former) encrypted data are now decrypted and you used the memory of just one byte array length. Now you can play the music from this byte array (steps 7 + 8).
Just for explanation, steps 1-3 are on server side and in steps 3+4 the transmission takes place.
This example uses the AES CTR mode as it provides the same length for input and output data.
In the end I’m comparing the byte arrays to prove that the decryption was successful for steps 6 (direct decryption) and 7 (decryption in chunks):
Decrypting 50mb data in chunks to avoid out of memory error https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom plaindata length: 52428810 cipherdata length: 52428810 decrypteddata length: 52428810 cipherdata parts in 16 byte long parts: 3276800 = rounds for decryption cipherdata moduluo 16 byte long parts: 10 + 1 round for rest/modulus cipherdata length: 52428810 (after decryption) plaindata equals decrypteddata: true plaindata equals cipherdata: true import javax.crypto.Cipher; import javax.crypto.spec.IvParameterSpec; import javax.crypto.spec.SecretKeySpec; import java.security.GeneralSecurityException; import java.security.SecureRandom; import java.util.Arrays; import java.util.Random; public class EncryptionCtrSo4 < public static void main(String[] args) throws GeneralSecurityException < System.out.println("Decrypting 50mb data in chunks to avoid out of memory error"); System.out.println("https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom"); /* * author michael fehr, http://javacrypto.bplaced.net * no licence applies, no warranty */ // === server side === // 1 create a 50 mb byte array (unencrypted) byte[] plaindata = new byte[(50 * 1024 * 1024 + 10)]; // fill array with (random) data Random random = new Random(); random.nextBytes(plaindata); // 2 encrypt the data with aes ctr mode, create random keys SecureRandom secureRandom = new SecureRandom(); byte[] key = new byte[32]; // 32 byte = 256 bit aes key secureRandom.nextBytes(key); byte[] iv = new byte[16]; // 16 byte = 128 bit secureRandom.nextBytes(iv); SecretKeySpec keySpecEnc = new SecretKeySpec(key, "AES"); IvParameterSpec ivParameterSpecEnc = new IvParameterSpec(iv); Cipher cipherEnc = Cipher.getInstance("AES/CTR/NoPadding"); cipherEnc.init(Cipher.ENCRYPT_MODE, keySpecEnc, ivParameterSpecEnc); byte[] cipherdata = cipherEnc.doFinal(plaindata); System.out.println("plaindata length: " + plaindata.length); System.out.println("cipherdata length: " + cipherdata.length); // 3 transfer the cipherdata to app // . // === app side === // 4 receive encrypted data from server // . // 5 decryption setup SecretKeySpec keySpecDec = new SecretKeySpec(key, "AES"); IvParameterSpec ivParameterSpecDec = new IvParameterSpec(iv); Cipher cipherDec = Cipher.getInstance("AES/CTR/NoPadding"); cipherDec.init(Cipher.DECRYPT_MODE, keySpecDec, ivParameterSpecDec); // 6 decryption in one run on high memory devices byte[] decrypteddata = cipherDec.doFinal(cipherdata); System.out.println("decrypteddata length: " + decrypteddata.length); // 7 decryption in chunks using the cipherdata byte array int cipherdataLength = cipherdata.length; int chunksize = 16; // should be a multiple of 16, minimum 16 byte[] decryptedPart = new byte[chunksize]; int parts16byteDiv = cipherdataLength / chunksize; int parts16byteMod = cipherdataLength % chunksize; System.out.println("cipherdata parts in " + chunksize + " byte long parts: " + parts16byteDiv + " = rounds for decryption"); System.out.println("cipherdata moduluo " + chunksize + " byte long parts: " + parts16byteMod + " + 1 round for rest/modulus"); for (int i = 0; i < parts16byteDiv; i++) < cipherDec.update(cipherdata, (i * chunksize), chunksize, decryptedPart); System.arraycopy(decryptedPart, 0, cipherdata, (i * chunksize), decryptedPart.length); >if (parts16byteMod > 0) < decryptedPart = new byte[parts16byteMod]; cipherDec.update(cipherdata, (parts16byteDiv * chunksize), parts16byteMod, decryptedPart); System.arraycopy(decryptedPart, 0, cipherdata, (parts16byteDiv * chunksize), decryptedPart.length); >System.out.println("cipherdata length: " + cipherdata.length + " (after decryption)"); // the cipherdata byte array is now decrypted ! // 8 use cipherdata (encrypted) or decrypteddata as input for exoplayer // compare ciphertext with decrypteddata in step 6 System.out.println("plaindata equals decrypteddata: " + Arrays.equals(plaindata, decrypteddata)); // check that (decrypted) cipherdata equals plaindata of step 7 System.out.println("plaindata equals cipherdata: " + Arrays.equals(plaindata, cipherdata)); > > How can I write a byte array to a file in Java?
As Sebastian Redl points out the most straight forward now java.nio.file.Files.write. Details for this can be found in the Reading, Writing, and Creating Files tutorial.
Old answer: FileOutputStream.write(byte[]) would be the most straight forward. What is the data you want to write?
The tutorials for Java IO system may be of some use to you.
KeyGenerator kgen = KeyGenerator.getInstance(«AES»); kgen.init(128); SecretKey key = kgen.generateKey(); byte[] encoded = key.getEncoded();
+1 for mentioning tutorials. (you’d received even a +2 if you mentioned www.google.com — OK, that was nasty, but it’s not rovers first question. )
You should wrap the FileOutputStream in a BufferedOutputStream as suggested by @JuanZe. Performance is much better.
KeyGenerator kgen = KeyGenerator.getInstance("AES"); kgen.init(128); SecretKey key = kgen.generateKey(); byte[] encoded = key.getEncoded(); FileOutputStream output = new FileOutputStream(new File("target-file")); IOUtils.write(encoded, output); @GauravAgarwal Not in her/his original question (which they should have updated!), but see the first and third comment here: stackoverflow.com/questions/1769776/…
As of Java 1.7, there’s a new way: java.nio.file.Files.write
import java.nio.file.Files; import java.nio.file.Paths; KeyGenerator kgen = KeyGenerator.getInstance("AES"); kgen.init(128); SecretKey key = kgen.generateKey(); byte[] encoded = key.getEncoded(); Files.write(Paths.get("target-file"), encoded); Java 1.7 also resolves the embarrassment that Kevin describes: reading a file is now:
byte[] data = Files.readAllBytes(Paths.get("source-file")); This is probably the recommended way nowadays. All the SecretKey stuff is not needed; just call Files.write() . Thanks Sebastian!
A commenter asked «why use a third-party library for this?» The answer is that it’s way too much of a pain to do it yourself. Here’s an example of how to properly do the inverse operation of reading a byte array from a file (sorry, this is just the code I had readily available, and it’s not like I want the asker to actually paste and use this code anyway):
public static byte[] toByteArray(File file) throws IOException < ByteArrayOutputStream out = new ByteArrayOutputStream(); boolean threw = true; InputStream in = new FileInputStream(file); try < byte[] buf = new byte[BUF_SIZE]; long total = 0; while (true) < int r = in.read(buf); if (r == -1) < break; >out.write(buf, 0, r); > threw = false; > finally < try < in.close(); >catch (IOException e) < if (threw) < log.warn("IOException thrown while closing", e); >else < throw e; >> > return out.toByteArray(); > Everyone ought to be thoroughly appalled by what a pain that is.
Use Good Libraries. I, unsurprisingly, recommend Guava’s Files.write(byte[], File).
How to convert Writer to Byte array
I need to write blob content to database. I am using a writer Object for this. That is:
OutputStreamWriter writer = getMyWriter(); I need to know how to convert this writer object to a byte array.? I need this byte array because , to create blob I pass byte array to the Hibernate entity. I have pasted the code for reference:
public void writeBolobSecuredContentToDb() throws Exception < OutputStreamWriter writer = getMyWriter(); formattedContentToWriter(writer); // Writing to Database UserPersonal secureData = UserPersonal() userPersonal.fromBytArray("How to convert my writer object to byte array?"); writer.flush(); writer.close(); >public OutputStreamWriter getMyWriter() < CipherOutputStream cos = null; try < cos = getEncrypttedCipherOutputStream(); return new OutputStreamWriter(cos); >catch (Exception e) < throw new IllegalStateException("Could not create cipher outputstream.", e); >> > //Hibernate entity public class UserPersonal implements java.io.Serializable public static Stock fromBytArray(byte[] content) What exactly do you want to put into database? From where do you get the data you want to put into db? What does getEncrypttedCipherOutputStream() do?
I need to put encrypted content to a ZIP and save as a blob in database. It is happening via getEncrypttedCipherOutputStream();
You don’t need to convert it to a byte array. You need to get an OutputStream from the Blob, and wrap your own CipherOutputStream , ZipOutputStream , OutputStreamWriter etc., around that, then write your data. You’re doing this upside down and back to front.